DKAN 1.13 Release notes

Read up on our latest release, 1.13! This is a huge release and we have new features and enhancements that will make using DKAN better than ever. Major new features include a Harvester, API improvements, new data previews, and DCAT-AP support.

Have questions or thoughts? Let us know on our public DKAN Repo in the issues queue or chat with us in our Gitter room.

What's new

DKAN Harvest

DKAN's new Harvest module gives DKAN administrators and site managers the ability to harvest datasets from other sources for their DKAN site. Built on the widely-used Drupal Migrate system, the Harvest module can fetch data and publish it to a DKAN site.

DKAN Harvest is fully configurable via the DKAN web interface. Harvest sources can be added like any other type of content, and data can be overridden or manipulated before being published. Migrations can be managed via an intuitive dashboard interface or triggered with a cron job.

Get more details on DKAN Harvest by reading its documentation pages.

DKAN Dataset REST API

Previous DKAN APIs have been read-only and focused on CKAN compatibility. A new RESTful API for DKAN datasets provides an endpoint that allows other applications to interact with DKAN, including creating and modifying datasets and resources.

Get more details on the Dataset REST API.

DCAT-AP Compliance

DKAN's DCAT RDF publishing has been reengineered to be more consistent with its other open-data standards implementation, and, more importantly, to implement the new DCAT-AP protocol. DKAN's fields can be easily mapped to DCAT-AP fields using the Open Data Schema Mapper. Now, DKAN meets US standards (Project Open Data) as well as European Union standards (DCAT-AP).

New Default Content

On a basic install of DKAN, the site came with only a few example Datasets and Resources. We added more default content in order to demonstrate the full capabilities on DKAN and give new users a better experience in testing out various DKAN features. The default content module is part of core DKAN, so all new installs of DKAN contain the content. Simply disabling the "DKAN Default Content" module will remove all the default content to give you a blank slate to start building.

What we've improved

The functionality isn't new, but it's working even better than before. These improvements make DKAN better on the whole including user experience, efficiency, and scale.

Site Managers can assign roles

The Site Manager role is part of a core group of user roles on DKAN. Site Managers have a high degree of control over the entire site, but are expected to be less technical than a full admin. Previously, it wasn't possible for Site Managers to assign roles to new users. By adding the RoleAssign module we have provided this more fine-grained permission.

Site Managers can add pages

One of the most basic content types on DKAN is a page. And though the content type is straightforward it can have implications for the structure and appearance of the site. Originally, Site Managers were not able to add or manage Pages; in DKAN 1.13 Site Managers now have this permission.

Even more permissions for Site Managers

• Theme Settings: Includes page elements and ability to add custom logo and favicon from "Site Configuration > Theme Settings" at /admin/appearance/settings
• Colorizer: Ability to create custom color schemes for a DKAN site from "Site Configuration > Colorizer" at /admin/appearance/colorizer
• Open Data Schema Mapper (ODSM): Ability to add, edit, and delete APIs and their mappings to DKAN fields from "Site Configuration > Open Data Schema Mapper" at /admin/config/services/odsm
• Menus: Permission to manage the main menu links by adding, editing, and deleting links from "Site Configuration > Menu" at /admin/structure/menu
• Topics Icons: When adding or editing terms in the Topics taxonomy at path /admin/structure/taxonomy/dkan_topics, site managers can choose an "Icon Type" of either "Font Icon" or "Image." If they choose "Font Icon," the edit form displays a list of available icons from which to choose. If they choose "Image," the form lets them upload an image.
• Enabling External Previews: Site Managers can enable previews so that site visitors can look at Resource contents with visualization tools, Carto and ArcGIS. To enable External Previews, go "DKAN > Data Previews" at /admin/dkan/dataset_preview
• DCAT and POD validation: Site Managers can select the settings for how Datasets are validated against Project Open Data and DCAT-AP standards.
• Site managers can confirm that the website's data.json feed is working correctly by using the POD online validator at https://labs.data.gov/dashboard/validate
• Site managers can confirm that the website's DCAT feed is working correctly from Site Configuration > Open Data Schema Mapper > DCAT Validation at /admin/config/services/odsm/validate/dcat or by using the DCAT online validator at http://www.dcat.be/validator/
• Order of Groups if using the featured groups block: Site Managers can arrange the order in which Groups appear in the featured group block from "DKAN > Featured Groups Sort Order" at /admin/dkan/featured-groups-sort-order
• Recline size configuration: Manage size constraints for Recline, which powers internal previews, from "DKAN > Recline Configuration" at /admin/dkan/recline

Open Data Federal Extras (ODFE) is now part of core DKAN

Project Open Data (POD) sets a standard for the information about datasets (metadata) included when the data is published. DKAN collects metadata with fields on the Dataset form when the data is published. For most agencies, the fields reflect basic requirements. However US federal agencies are required to provide additional information about the data published. The extra fields appear as part of the Dataset form with our ODFE module.

We've moved this module into core DKAN so that it's part of every install, though not enabled by default. When enabled, the ODFE module makes federal agencies compliant with POD standards.

Get more details on Federal Extras on DKAN.

In-place Editor for Site Managers + look and feel

Previously, the In-place Editor (IPE, an interface for changing page layouts) for Site Managers included several options that weren't appropriate for their level of permissions. This UI was confusing, so we improved the interface to only include options that are relevant to non-admin user roles. We also improved the overall look and feel of the IPE to be more user-friendly.

Improved Data Previews

The Data Previews feature is designed to let site visitors take a sneak peek at the content of a Resource before downloading the file. In this release, we've improved Data Previews to support more file formats beyond CSV and XLS. Now previews also support JSON, geojson, XML, ArcGIS REST, WMS, images, PDF, and ZIP data formats.

Improvements to Data Previews also offer better support for Resources that are hosted remotely. Previously, Resources that were linked to a web source could only be previewed if they were first imported into the DKAN Datastore (which only supported CSV files). In DKAN 1.13, linked Resources can be previewed with the Data Previews feature.

Improved Datastore API

The DKAN Datastore API makes it possible to query for contents of Resources uploaded to the Datastore as detailed as a cell within a table. The improved Datastore API is enhanced to open greater possibilities of requesting complex queries from contents within a Datastore on a DKAN site as well as multiple queries in a single request.

Get more details on the Datastore API

New Dataset fields added

A number of metadata fields that are common requirements for open data standards like Project Open Data and DCAT-AP have been added to the basic dataset content type in DKAN. These fields are:

• Data Standard
• Data Dictionary Type
• Homepage URL
• Data Standard (conformsTo)
• Rights
• Language

The following fields are also included but hidden. This allows the harvester to ingest metadata with these fields and present them in data.json but they are not yet supported in the UI for locally-created datasets:

• Collection (isPartOf)
• Theme

See the Project Open Data schema for more information on these fields.

NuBoot Radix now in core DKAN

Previously, the NuBoot Radix theme on DKAN was not part of the core repository. The old Omega-based theme and its tools (the Delta module) have been removed, and NuBoot Radix is no longer maintained in a separate repository. Legacy themes will not be maintained or shipped with DKAN and require manual installation if you want to use the Omega theme. Alternatively, you can make customizations to the default theme.

Additionally, the admin theme setting is set to use the default theme rather than the NuBoot Radix theme. Because the Colorizer tool requires both the admin and default to be the same, setting the admin theme setting to use the default theme enables Colorizer to work normally for both the NuBoot subtheme and other custom subthemes without admins needing to manually override this setting.

Added POD-based validation on Datasets

A new option was added on DKAN Dataset Forms at /admin/dkan/dataset_forms in order to enable form validation based on Project Open Data. If "Validate dataset form according to Project Open Data" is enabled, then all POD required fields will be also required in DKAN. See the POD schema for details of the validation standards: https://project-open-data.cio.gov/v1.1/schema/

Minor improvements

• We added more frequency update options to the Dataset form to comply with Project Open Data standards
• Added icons to the Topics drop-down menu.
• Renamed the default HTML text format from "HTML" to "Markdown HTML." This is also reflected in the UI when adding new content.
• Made URLs in the Additional info section of Datasets display properly and made them clickable
• Removed the Conditional Fields module from DKAN, which was adding bloat and creating lots of unwanted artifacts in features, and only used by two fields. A few lines of Form API code was enough to reproduce the functionality.
• Several improvements were made to enhance accessibility and 508 compliance on DKAN including labels, alt text in the UI, table headers and more.
• The group field has been simplified in the dataset node form and removed from the resource node form (resources always inherit their datasets' groups.)

What we fixed

Some DKAN features weren't working as expected and causing issues. We fixed those, so you can keep using DKAN as expected.

• Data proxy for Google spreadsheets was broken. This meant that the data in a Resource could not be parsed properly, and these files could not be used to generate Charts with the DKAN Visualization Entity. We fixed this so Resources that are not internal to DKAN can be used to create a Chart on DKAN.
• The DKAN API endpoints for JSON and RDF outputs did not generate the proper file when called. We fixed this so that the JSON endpoint works normally. With our new DCAT-AP compliance feature, the RDF output will generate an XML file.
• Several bugs in the Visualization entity resulted in a frustrating user experience when attempting to create a Chart. A number of fixes were made to result in the expected behavior of the Chart.
• Only certain roles had permission to use the "Markdown HTML" text format, which resulted in many users unable to edit text to content which had been created using that format. We fixed that so now all roles can add text to a text box regardless of the text format.
• The toolbar that appears at the top of the text area when using the "Markdown" text format used to include some buttons for styling options that DKAN did not actually support, which created confusion. We've removed the unsupported buttons from the toolbar so it only contains supported options (headings, bold, italic, lists, blockquote, link, image, line break).

Upgrading to the latest version

Removing EVA module

WARNING: The EVA is removed with this release. If you are using EVA in your project you need to add it to your sites/all/modules folder.

One view previously used EVA, user_profile_search. You can avoid an error by reverting the feature dkan_sitewide_user. Note that we recommend reverting all features drush fra -y for the upgrade.

Features Reverts

The following commands should be run to for the upgrade:

ahoy drush rr
ahoy drush fr dkan_dataset_content_types -y
ahoy drush fr dkan_permissions -y
If ODFE is enabled then: ahoy drush fr open_data_federal_extras -y
If ODFE is enabled then: rm -rf sites/all/modules/open_data_federal_extras
ahoy drush fr open_data_schema_map_dkan -y
ahoy drush updatedb -y
ahoy drush en dkan_ipe -y
ahoy drush en dkan_harvest_dashboard -y
ahoy drush en menu_admin_per_menu -y
ahoy drush fra -y
ahoy drush rr

New Documentation location

We have moved DKAN's technical documentation from the Drupal-based site where it used to live to the /docs folder within the main DKAN repo. Documentation changes will now be part of code commits and pull requests. The documentation is built on ReadTheDocs, but lives at the same URL as the old site: http://docs.getdkan.com/

Special Notes

Front Page - Upgrade Instructions

Front page configuration has been removed from features and the dkan_sitewide_demo_front feature has been deprecated. To save existing front page configuration, run the following command drush php-eval "dkan_sitewide_convert_panel_page('<page-name>', TRUE);" This will convert the front page to a panelized node.

Also note that if you use any of the default DKAN blocks for your front page, some may dissapear after the upgrade. A user with the administrator role should be able to restore them by editing the panel and finding them in the Miscellaneous list.

Added Modules

• DKAN Harvest
• DKAN Harvest Dashboard
• DKAN IPE
• Menu Admin per Menu

Removed Modules - The following modules were removed from the code base and upgrading will remove the files and db tables associated with them.

• Conditional Fields
• Entity RDF
• RDF UI
• RDF Extensions
• Delta
• EVA

Removed Themes

• Omega

Consolidation - External repos have been merged into DKAN core. This means that dkan_dataset, dkan_workflow, dkan_datastore and nuboot_radix are all included in the main dkan profile repo.

To remove newly untracked files run:

git clean -f

Known issues

• Site managers do not have permission to see the Project Open Data validator screen, which tells them if their site is currently producing a valid data.json. This will be fixed in 1.13.1. The equivalant screen is availabe for DCAT-AP, under the configuration menu. If you need to check your site's data.json feed in the meantime, you can use Data.gov's online POD validator at https://labs.data.gov/dashboard/validate.