[META] DCAT v3 support

[amercader]

DCAT v3 support in CKAN

Summary

The aim is for CKAN to provide DCAT 3 support out of the box with minimal configuration, both as a metadata provider and as a consumer, providing comprehensive base implementations that can be adapted to each site needs.

Note

This is an evolving plan, it can change its shape and scope as time progresses 🦠

Primary goals

• Support for DCAT 3 based Application Profiles (both as Provider and Consumer):
  • For European portals: DCAT-AP 3.0.0
  • For US portals: DCAT-US 3.0.0
• Provide base implementations of both profiles that site maintainers can easily adapt to their needs
• Provide built-in support for DCAT entities not directly modelled in CKAN, like Dataset Series or Data Services (core or extension)

Secondary goals

We have also identified other areas that could be explored as part of the work on the main goals above. While it would be great to be able to focus on all them, this will depend on existing availability:

• Create a pre-configured CKAN distribution with all necessary extension and configuration settings to run a DCAT-base site
• Explore options for loss-less harvesting of arbitrary metadata fields for the CKAN-in-the-middle ¹ use case
• Close integration with ckanext-spatial for spatial metadata fields (spatial indexing, map previews, etc)

Functionality:

• New processors (parsers and serializers) for DCAT-AP 3 and DCAT-US 3 to map DCAT to CKAN metadata and vice versa
• Out of the box schemas for Datasets and Resources that support all properties defined in DCAT-AP and DCAT-US, as well as controlled vocabularies required
• Pre-configured scheming presets and widgets for complex fields like repeating fields, date ranges, etc
• Finished and documented support for multilingual DCAT fields
• UI and underlying API actions to manage new entities (e.g. for Dataset Series: defining the dataset, adding items to the series, etc)

Use cases:

• Publishers that are required to publish their metadata following DCAT-AP 3 or DCAT-US 3 can use CKAN out of the box
• CKAN sites that need to adhere to national or regional variations can easily modify the base profiles and schemas to adapt them to their needs
• Portals that aggregate remote metadata in CKAN can harvest or import different sources that follow DCAT 3 based standards (alongside previous versions of DCAT and other formats)

Work items

Note

These need to be split and expanded in individual issues

Getting ready

• New version of ckanext-dcat that includes updated requirements, support for CKAN 2.11 and if possible multilingual support. This will be the base for all DCAT 3 support
• Scheming implementation of the currently supported DCAT-AP 2.x (#56)

Core Dataset and Distribution properties

• Adapt parsers to consume/expose new fields for Datasets and Resources (Distributions) in DCAT-AP 3
• Create schemas for DCAT-AP 3 + UI widgets (repeating, etc)
• Integrate SHACL validation to prove compliance (DCAT-AP)
• Repeat process for DCAT-US

Dataset Series

• Create a specification and design the feature
• Implement it at the CKAN level
• Expose it / consume it as DCAT metadata

Data Service

• Create a specification and design the feature
• Implement it at the CKAN level
• Expose it / consume it as DCAT metadata
  Other

Previous CKAN - DCAT 3 discussions

• Support for DCAT 3 ckan#7855
• Support for DCAT 3 #257
• Solution: DCAT v3 extension for CKAN ckan#7896
• Problem: It's possible to use graph database as the main data storage or to efficiently solve workflows with graph databases ckan#7489

Footnotes

1. Describes a scenario where CKAN harvests metadata from a lower level site, and in turn it exposes its metadata to a higher level aggregator portal ↩

[ivbeg]

Hi! I represent CKAN and other data catalogs aggregator Dateno (dateno.io) and several CKAN based data catalogues.
My thoughts about DCAT 3 implementation:

1. DCAT 2 and/or 3 should be enabled by default. It's a good practice to provide bulk metadata access as DCAT instead of CKAN regular API. I would say that it's much easier to collect metadata from DCAT-first data catalogs.
2. There are some variations allowed by DCAT 2 and 3 about dataset theme and some other fields. It would be great if CKAN team could provide global reference metadata or if it could be possible to configure dictionaries during CKAN configuration.
3. Looking further it would be great if CKAN will be much more customizable not just ot support DCAT 3, but other metadata standards as well.
4. DCAT 3 introduces several important fields related to dataset version management. I am sure that most CKAN users don't aware much about it. I think that some educational resources needed about that.
5. DCAT 3 doesn't break DCAT 2. Please keep providing export of CKAN data with both DCAT 2 and DCAT 3 standards. It will not break existing parsers.