ckanext-schemingdcat. LOD/INSPIRE metadata enhancement for ckanext-scheming

Overview • Installation • Configuration • Schemas • Harvesters • Running the Tests

Overview

This CKAN extension provides functions and templates specifically designed to extend ckanext-scheming and includes DCAT and Harvest enhancements to adapt CKAN Schema to GeoDCAT-AP.

Warning

Requires mjanez/ckanext-dcat, ckan/ckanext-scheming and ckan/ckanext-spatial to work properly.

Tip

It is recommended to use with: ckan-docker deployment or only use ckan-pycsw to deploy a CSW Catalog.

Enhancements:

• Could use schemas for ckanext-scheming in the plugin like CKAN GeoDCAT-AP schema
• Improve the search functionality in CKAN for custom schemas. It uses the fields defined in a scheming file to provide a set of tools to use these fields for scheming, and a way to include icons in their labels when displaying them. More info: ckanext-schemingdcat
• Add improved harvesters for custom metadata schemas integrated with ckanext-harvest in CKAN using mjanez/ckan-ogc.
• Add Metadata downloads for Linked Open Data formats (mjanez/ckanext-dcat) and Geospatial Metadata (ISO 19139, Dublin Core, etc. with mjanez/ckan-pycsw)
• Add custom i18n translations to datasets, groups, organizations in schemas, e.g: GeoDCAT-AP (ES).¹
• Add a set of useful helpers and templates to be used with Metadata Schemas.
• Update the base theme of CKAN to use with the enhancements of this extension.
• Modern UI inspired on datopian/ckanext-datopian.
• LOD/OGC Endpoints based on avalaible profiles (DCAT) and CSW capabilities with mjanez/ckan-pycsw.

Requirements

This plugin is compatible with CKAN 2.9 or later and needs the following plugins to work properly:

# Install latest stable release of:
## ckan/ckanext-scheming: https://github.com/ckan/ckanext-scheming/tags (e.g. release-3.0.0)
pip install -e git+https://github.com/ckan/ckanext-scheming.git@release-3.0.0#egg=ckanext-scheming

## mjanez/ckanext-dcat: https://github.com/mjanez/ckanext-dcat/tags (e.g. 1.2.0-geodcatap)
pip install -e git+https://github.com/mjanez/ckanext-dcat.git@1.2.0-geodcatap#egg=ckanext-dcat
pip install -r https://raw.githubusercontent.com/mjanez/ckanext-dcat/master/requirements.txt

## ckan/ckckanext-spatial: https://github.com/ckan/ckanext-spatial/tags (e.g. v2.1.1)
pip install -e git++https://github.com/ckan/ckanext-spatial.git@v2.1.1/#egg=ckanext-spatial#egg=ckanext-spatial
pip install -r https://raw.githubusercontent.com/ckan/ckanext-spatial/v2.1.1/requirements.txt

## ckan/ckckanext-harvest: https://github.com/ckan/ckanext-harvest/tags (e.g. v1.5.6)
pip install -e git++https://github.com/ckan/ckanext-harvest.git@v1.5.6#egg=ckanext-spatial
pip install -r https://raw.githubusercontent.com/ckan/ckanext-harvest/v1.5.6/requirements.txt

Installation

cd $CKAN_VENV/src/

# Install the scheming_dataset plugin
pip install -e "git+https://github.com/ckan/ckanext-schemingdcat.git#egg=ckanext-schemingdcat"

Configuration

Set the plugin:

# Add the plugin to the list of plugins
ckan.plugins = ... spatial_metadata ... dcat ... schemingdcat

Warning

When using schemingdcat extension,scheming should not appear in the list of plugins loaded in CKAN. But dcat and spatial should.

Scheming DCAT

Set the schemas you want to use with configuration options:

# Each of the plugins is optional depending on your use
ckan.plugins = schemingdcat_datasets schemingdcat_groups schemingdcat_organizations

To use CSW Endpoint in ckanext-schemingdcat:

schemingdcat.geometadata_base_uri = http://localhost:81/csw
ckanext.dcat.base_uri = http://localhost:81/catalog

To use custom schemas in ckanext-scheming:

# module-path:file to schemas being used
scheming.dataset_schemas = ckanext.schemingdcat:schemas/geodcatap/geodcatap_dataset.yaml
scheming.group_schemas = ckanext.schemingdcat:schemas/geodcatap/geodcatap_group.json
scheming.organization_schemas = ckanext.schemingdcat:schemas/geodcatap/geodcatap_org.json

#   URLs may also be used, e.g:
#
# scheming.dataset_schemas = http://example.com/spatialx_schema.yaml

#   Preset files may be included as well. The default preset setting is:
scheming.presets = ckanext.schemingdcat:schemas/geodcatap/geodcatap_presets.json

#   The is_fallback setting may be changed as well. Defaults to false:
scheming.dataset_fallback = false

Harvest

Add the custom Harvesters to the list of plugins as you need:

ckan.plugins = ... spatial_metadata ... dcat ... schemingdcat ... harvest ... schemingdcat_ckan_harvester schemingdcat_csw_harvester ...

Endpoints

You can update the endpoints.yaml file to add your custom OGC/LOD endpoints, only has 2 types of endpoints: lod and ogc, and the profile avalaible in ckanext-dcat Preferably between 4 and 8.

Examples:

• LOD endpoint: A Linked Open Data endpoint is a DCAT endpoint that provides access to RDF data. More information about the catalogue endpoint, how to use the endpoint, (e.g. https://{ckan-instance-host}/catalog.{format}?[page={page}]&[modified_since={date}]&[profiles={profile1},{profile2}]&[q={query}]&[fq={filter query}], and more at ckanext-dcat

    - name: euro_dcat_ap_2_rdf
      display_name: RDF DCAT-AP
      type: lod
      format: rdf
      image_display_url: /images/icons/endpoints/euro_dcat_ap_2.svg
      description: RDF DCAT-AP Endpoint for european data portals.
      profile: euro_dcat_ap_2
      profile_label: DCAT-AP
      version: null

• OGC Endpoint: An OGC CSW endpoint provides a standards-based interface to discover, browse, and query metadata about spatial datasets and data services. More info about the endpoint at OGC: Catalogue Servicestandard/cat/

    - name: csw_inspire
      display_name: CSW INSPIRE 2.0.2
      type: ogc
      format: xml
      image_display_url: /images/icons/endpoints/csw_inspire.svg
      description: OGC-INSPIRE Endpoint for spatial metadata.
      profile: spain_dcat
      profile_label: INSPIRE
      version: 2.0.2

Facet Scheming

To configure facets, there are no mandatory sets in the config file for this extension. The following sets can be used:

schemingdcat.facet_list: [list of fields]      # List of fields in scheming file to use to faceting. Use ckan defaults if not provided.
schemingdcat.default_facet_operator: [AND|OR]  # OR if not defined

 schemingdcat.icons_dir: (dir)                  # images/icons if not defined

As an example for facet list, we could suggest:

schemingdcat.facet_list = "theme groups theme_es dcat_type owner_org res_format publisher_name publisher_type frequency tags tag_uri conforms_to spatial_uri"

The same custom fields for faceting can be used when browsing organizations and groups data:

schemingdcat.organization_custom_facets = true
schemingdcat.group_custom_facets = true

This two last settings are not mandatory. You can omit one or both (or set them to false), and the default fields for faceting will be used instead.

Facet Scheming integration with Solr

1. Clear the index in solr:

   ckan -c [route to your .ini ckan config file] search-index clear

2. Modify the schema file on Solr (schema or managed schema) to add the multivalued fields added in the scheming extension used for faceting. You can add any field defined in the schema file used in the ckanext-scheming extension that you want to use for faceting. You must define each field with these parameters:

   • type: string - to avoid split the text in tokens, each individually "faceted".
   • uninvertible: false - as recomended by solr´s documentation
   • docValues: true - to ease recovering faceted resources
   • indexed: true - to let ckan recover resources under this facet
   • stored: true - to let the value to be recovered by queries
   • multiValued: well... it depends on if it is a multivalued field (several values for one resource) or a regular field (just one value). Use "true" or "false" respectively.

   E.g. ckanext-iepnb extension are ready to use these multivalued fields. You have to add this configuration fragment to solr schema in order to use them:

   <!-- Extra fields -->
     <field name="tag_uri" type="string" uninvertible="false" docValues="true" indexed="true" stored="true" multiValued="true"/>
     <field name="conforms_to" type="string" uninvertible="false" docValues="true" indexed="true" stored="true" multiValued="true"/>
     <field name="lineage_source" type="string" uninvertible="false" docValues="true" indexed="true" stored="true" multiValued="true"/>
     <field name="lineage_process_steps" type="string" uninvertible="false" docValues="true" indexed="true" stored="true" multiValued="true"/>
     <field name="reference" type="string" uninvertible="false" docValues="true" indexed="true" stored="true" multiValued="true"/>
     <field name="theme" type="string" uninvertible="false" docValues="true" indexed="true" stored="true" multiValued="true"/>
     <field name="theme_es" type="string" uninvertible="false" docValues="true" multiValued="true" indexed="true" stored="true"/>
     <field name="metadata_profile" type="string" uninvertible="false" docValues="true" multiValued="true" indexed="true" stored="true"/>
     <field name="resource_relation" type="string" uninvertible="false" docValues="true" indexed="true" stored="true" multiValued="true"/>

   [!NOTE] You can ommit any field you're not going to use for faceting, but the best policy could be to add all values at the beginning.

   The extra fields depend on your schema

   Be sure to restart Solr after modify the schema.

3. Restart CKAN.

4. Reindex solr index:

   ckan -c [route to your .ini ckan config file] search-index rebuild-fast

   Sometimes solr can issue an error while reindexing. In that case I'd try to restart solr, delete index ("search-index clear"), restart solr, rebuild index, and restart solr again.

   Ckan needs to "fix" multivalued fields to be able to recover values correctly for faceting, so this step must be done in order to use faceting with multivalued fields.

Icons

Icons for each field option in the scheming file can be set in multiple ways:

• Set a root directory path for icons for each field using the icons_dir key in the scheming file.
• If icons_dir is not defined, the directory path is guessed starting from the value provided for the schemingdcat.icons_dir parameter in the CKAN config file, adding the name of the field as an additional step to the path (public/images/icons/{field_name).
• For each option, use the icon setting to provide the last steps of the icon path from the field's root path defined before. This value may be just a file name or include a path to add to the icon's root directory.
• If icon is not used, a directory and file name are guessed from the option's value.
• Icons files are tested for existence when using schemingdcat_schema_icon function to get them. If the file doesn't exist, the function returns None. Icons can be provided by any CKAN extension in its public directory.
• Set a default icon for a field using the default_icon setting in the scheming file. You can get it using schemingdcat_schema_get_default_icon function, and it is your duty to decide when and where to get and use it in a template.

New theme

Update the base theme of CKAN to use with the enhancements of this extension.

Schemas

With this plugin, you can customize the group, organization, and dataset entities in CKAN. Adding and enabling a schema will modify the forms used to update and create each entity, indicated by the respective type property at the root level. Such as group_type, organization_type, and dataset_type. Non-default types are supported properly as is indicated throughout the examples.

Are available to use with this extension a number of custom schema, more info: schemas/README.md

Schema Enhancements: We've made several improvements to our schema to provide a better metadata and metadata group management. Here are some of the key changes:

• Form Groups: We've introduced the use of form_groups and improve form_pages in our schemas. This allows us to group related fields into the same form, making it easier to navigate and manage metadata.
• Metadata Management Improvements: We've improved how we manage metadata in our schema. It's now easier to add, remove, and modify metadata, allowing us to keep our data more organized and accessible.
• Metadata Group Updates: We've made changes to how we handle metadata groups (form_groups). It's now easier to group related metadata, helping us keep our data more organized and making it easier to find specific information.

For more details on these enhancements check Form Groups documentation, please refer to the schema files in ckanext/schemingdcat/schemas.

GeoDCAT-AP (ES)

schemas/geodcatp_es with specific extensions for spatial data and GeoDCAT-AP/INSPIRE metadata profiles.

Note

RDF to CKAN dataset mapping: GeoDCAT-AP (ES) to CKAN

DCAT

schemas/dcat based on: DCAT.

Note

RDF to CKAN dataset mapping: DCAT to CKAN

DCAT-AP (EU)

schemas/dcatap based on: DCAT-AP for the european context.

Note

RDF to CKAN dataset mapping: DCAT-AP (EU) to CKAN

GeoDCAT-AP (EU)

schemas/geodcatap based on: GeoDCAT-AP for the european context.

Note

RDF to CKAN dataset mapping: GeoDCAT-AP (EU) to CKAN

Form Groups

Form groups are a way to group related fields together in the same form. This makes it easier to navigate and manage metadata. A form group is defined with the following elements:

• form_group_id: A unique identifier for the form group. For example, contact.
• label: A human-readable label for the form group. This can be provided in multiple languages. For example:

  label: 
    en: Contact information 
    es: Información de contacto

• fa_icon: An optional Font Awesome icon that can be used to visually represent the form group. For example, fa-address-book.

Here is an example of a form group definition:

form_group_id: contact 
label: 
  en: Contact information 
  es: Información de contacto 
fa_icon: fa-address-book

Adding Fields to Form Groups

Fields can be added to a form group by specifying the form_group_id in the field definition. The form_group_id should match the form_group_id of the form group that the field should be part of.

Here is an example of a field that is part of the general_info form group:

field_name: owner_org
label:
  en: Organization
  es: Organización
required: True
help_text:
  en: Entity (organisation) responsible for making the Dataset available.
  es: Entidad (organización) responsable de publicar el conjunto de datos.
preset: dataset_organization
form_group_id: general_info

In this example, the owner_org field will be part of the general_info form group.

Harvesters

Basic using

In production, when gather and consumer processes are running, the following command are used to start and stop the background processes:

• ckan harvester run: Starts any harvest jobs that have been created by putting them onto the gather queue. Also checks running jobs - if finished it changes their status to Finished.

To testing harvesters in development, you can use the following command:

• ckan harvester run-test {source-id/name}: This does all the stages of the harvest (creates job, gather, fetch, import) without involving the web UI or the queue backends. This is useful for testing a harvester without having to fire up gather/fetch_consumer processes, as is done in production.

  [!WARNING] After running the run-test command, you should stop all background processes for gather and consumer to avoid conflicts.

Scheming DCAT CKAN Harvester: CKAN Harvester for custom schemas

The plugin includes a harvester for remote CKAN instances using the custom schemas provided by schemingdcat and ckanext-scheming. This harvester is a subclass of the CKAN Harvester provided by ckanext-harvest and is designed to work with the schemingdcat plugin to provide a more versatile and customizable harvester for CKAN instances.

To use it, you need to add the schemingdcat_ckan_harvester plugin to your options file:

  ckan.plugins = harvest schemingdcat schemingdcat_datasets ... schemingdcat_ckan_harvester

The Scheming DCAT CKAN Harvester supports the same configuration options as the CKAN Harvester, plus the following additional options:

• dataset_field_mapping: Mapping field names from local to remote instance. This is a dictionary with the following keys:
  • local_field_name: The field_name in the local instance, check the schema: http://ckan-instance/catalog/api/3/action/scheming_dataset_schema_show?type=dataset.
  • remote_field_name: The field_name in the remote instance. If the remote schema does not contain translations in the fields 'title_translated':{'en': 'example_record', 'en': 'example_record'}, but you want the values to be updated on load, you can define it as a dictionary where the keys refer to the names of the fields in the remote record, e.g. {'en': 'title', 'en': 'title-en'}.
• resource_field_mapping: Mapping field names from local to remote instance. This is a dictionary with the following keys:
  • local_field_name: The field_name in the local instance, check the schema: http://ckan-instance/catalog/api/3/action/scheming_dataset_schema_show?type=dataset.
  • remote_field_name: The field_name in the remote instance. If the remote schema does not contain translations in the fields 'title_translated':{'en': 'example_record', 'en': 'example_record'}, but you want the values to be updated on load, you can define it as a dictionary where the keys refer to the names of the fields in the remote record, e.g. {'en': 'title', 'en': 'title-en'}.
• schema: The name of the schema to use for the harvested datasets. This is the schema_name as defined in the scheming file. The remote and local instances must have the same dataset schema. If not provided, the local instance schema will be used.
• allow_harvest_datasets: If true, the harvester will create new records even if the package type is from the harvest source. If false, the harvester will only create records that originate from the instance. Default is false.
• remote_orgs: [WIP]. Only only_local.
• remote_groups: [WIP]. Only only_local.

And example configuration might look like this:

    {
    "api_version": 2,
    "default_tags": [{"name": "inspire"}, {"name": "geodcatap"}],
    "default_groups": ["transportation", "hb"],
    "default_extras": {"encoding":"utf8", "harvest_description":"Harvesting from Sample Catalog", "harvest_url": "{harvest_source_url}/dataset/{dataset_id}"},
    "organizations_filter_include": ["remote-organization"],
    "groups_filter_include":[],
    "override_extras":false,
    "user":"harverster-user",
    "api_key":"<REMOTE_API_KEY>",
    "read_only": true,
    "remote_groups": "only_local",
    "remote_orgs": "only_local",
    "schema": "geodcatap",
    "allow_harvest_datasets":false,
    "dataset_field_mapping": {"notes_translated": {"es": "notes-es"}, "provenance": {"es": "provenance"}, "purpose": {"es": "purpose"}, "title_translated": {"es": "title"}, "maintainer": "maintainer_name"},
    }

Scheming DCAT CSW INSPIRE Harvester

A harvester for remote CSW catalogues using the INSPIRE ISO 19139 metadata profile. This harvester is a subclass of the CSW Harvester provided by ckanext-spatial and is designed to work with the schemingdcat plugin to provide a more versatile and customizable harvester for CSW endpoints and GeoDCAT-AP CKAN instances.

To use it, you need to add the schemingdcat_csw_harvester plugin to your options file:

  ckan.plugins = harvest schemingdcat schemingdcat_datasets ... schemingdcat_csw_harvester

==#TODO:==

Remote XLS/XLSX Metadata Batch Harvester

A harvester for remote Excel files with Metadata records. This harvester is a subclass of the Scheming DCAT Base Harvester provided by ckanext-schemingdcat to provide a more versatile and customizable harvester for Excel files that have metadata records in them.

To use it, you need to add the schemingdcat_xls_harvester plugin to your options file:

ckan.plugins = harvest schemingdcat schemingdcat_datasets ... schemingdcat_xls_harvester

The Remote XLS/XLSX Metadata Batch Harvester supports the following options:

• storage_type - Mandatory: The type of storage to use for the harvested datasets as onedrive or gspread. Default is onedrive.
• dataset_sheet - Mandatory: The name of the sheet in the Excel file that contains the dataset records.
• dataset_field_mapping: Mapping field names from local to remote instance. This is a dictionary with the following keys:
  • local_field_name: The field_name in the local instance, check the schema: http://ckan-instance/catalog/api/3/action/scheming_dataset_schema_show?type=dataset.
  • remote_field_name: The field_name in the remote instance. If the remote schema does not contain translations in the fields 'title_translated':{'en': 'example_record', 'en': 'example_record'}, but you want the values to be updated on load, you can define it as a dictionary where the keys refer to the names of the fields in the remote record, e.g. {'en': 'title', 'en': 'title-en'}.
• resource_field_mapping: Mapping field names from local to remote instance. This is a dictionary with the following keys:
  • local_field_name: The field_name in the local instance, check the schema: http://ckan-instance/catalog/api/3/action/scheming_dataset_schema_show?type=dataset.
  • remote_field_name: The field_name in the remote instance. If the remote schema does not contain translations in the fields 'title_translated':{'en': 'example_record', 'en': 'example_record'}, but you want the values to be updated on load, you can define it as a dictionary where the keys refer to the names of the fields in the remote record, e.g. {'en': 'title', 'en': 'title-en'}.
• auth: If the remote file is private and requires authentication, the auth parameter should be used to provide the authentication credentials. Default is False.
  • credentials: If auth is True, the credentials parameter should be used to provide the authentication credentials. The credentials depends on the storage_type used.
    • For onedrive: The credentials parameter should be a dictionary with the following keys: username: A string representing the username. password: A string representing the password.
    • For gspread or gdrive: The credentials parameter should be a string containing the credentials in JSON format. You can obtain the credentials by following the instructions provided in the Google Workspace documentation.
• distribution_sheet: The name of the sheet in the Excel file that contains the distribution records. If not provided, the harvester will only create records for the dataset sheet.
• datadictionary_sheet: The name of the sheet in the Excel file that contains the data dictionary records. If not provided, the harvester will only create records for the dataset sheet.
• api_version: You can force the harvester to use either version 1 or 2 of the CKAN API. Default is 2.
• default_tags: A list of tags that will be added to all harvested datasets. Tags don't need to previously exist. This field takes a list of tag dicts which allows you to optionally specify a vocabulary. Default is [].
• default_groups: A list of group IDs or names to which the harvested datasets will be added to. The groups must exist in the local instance. Default is [].
• default_extras: A dictionary of key value pairs that will be added to extras of the harvested datasets. You can use the following replacement strings, that will be replaced before creating or updating the datasets:
  • {dataset_id}
  • {harvest_source_id}
  • {harvest_source_url} Will be stripped of trailing forward slashes (/)
  • {harvest_source_title}
  • {harvest_job_id}
  • {harvest_object_id}
• override_extras: Assign default extras even if they already exist in the remote dataset. Default is False (only non existing extras are added).
• user: User who will run the harvesting process. Please note that this user needs to have permission for creating packages, and if default groups were defined, the user must have permission to assign packages to these groups.
• read_only: Create harvested packages in read-only mode. Only the user who performed the harvest (the one defined in the previous setting or the 'harvest' sysadmin) will be able to edit and administer the packages created from this harvesting source. Logged in users and visitors will be only able to read them.
• force_all: By default, after the first harvesting, the harvester will gather only the modified packages from the remote site since the last harvesting Setting this property to true will force the harvester to gather all remote packages regardless of the modification date. Default is False.
• clean_tags: By default, tags are stripped of accent characters, spaces and capital letters for display. Setting this option to False will keep the original tag names. Default is True.
• source_date_format: By default the harvester uses dateutil to parse the date, but if the date format of the strings is particularly different you can use this parameter to specify the format, e.g. %d/%m/%Y. Accepted formats are: COMMON_DATE_FORMATS

And example configuration might look like this:

    {
    "auth": false,
    "api_version": 2,
    "dataset_sheet": "Datasets",
    "dataset_field_mapping": {"notes_translated": {"en": "notes-en", "es": "notes-es"}, "provenance": {"en": "provenance-en", "es": "provenance-es"}, "purpose": {"en": "purpose-en", "es": "purpose-es"}, "title_translated": {"en": "title-en", "es": "title-es"}, "maintainer": "maintainer_name"},
    "distribution_sheet": "Distributions",
    "datadictionary_sheet": "DataDictionary",
    "default_tags": [{"name": "inspire"}, {"name": "geodcatap"}],
    "default_groups": ["transportation", "hb"],
    "default_extras": {"encoding":"utf8", "harvest_url": "{harvest_source_url}/dataset/{dataset_id}"},
    "storage_type": "gspread",
    "user":"harverster-user",
    "read_only": true,
    "source_date_format": "%d/%m/%Y"
    }

Running the Tests

To run the tests:

pytest --ckan-ini=test.ini ckanext/schemingdcat/tests

Footnotes

1. An improvement to [ckanext-fluent] (https://github.com/ckan/ckanext-fluent) to allow more versatility in multilingual schema creation and metadata validation. ↩