💺🥩🍷 First-class Files

[wardi]

Uploaded files in CKAN are limited to 0 or 1 file attached to only groups or resources.

The group or resource model stores a reference to the file with a plain text column that can be updated like other metadata values. Resources can store the length, hash and format of a file uploaded, but these are metadata fields free for users to update (or not) that aren't durably linked to the file itself.

Uploaded files can leak, staying on the underlying storage and costing money even though there is no longer any way to reach them from the CKAN site.

There is no shared way to represent files that aren't yet attached to a group or resource, e.g:

• large file uploads in-progress with the ability to resume https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3/ManagedUpload.html
• files created before creating the dataset/resource metadata Resource-first "Add Dataset" workflow #6689
• files that need to be checked for validity before being accepted https://github.com/qld-gov-au/ckanext-validation

It's not possible to attach multiple files to a resource even when they represent the same data. This would be very useful for:

• sites like https://open.toronto.ca/catalogue/ where geospatial data is automatically converted into multiple formats and projections
• data that can be split across files like the Parquet format https://github.com/apache/parquet-format
• data generated by analyzing the resource Metadata resources: A special resource type for storing metadata #7856

Model solution

Let's create a model for uploaded files in CKAN that can be linked to resources or groups or anything else that a site might need.

Files would have:

• owner type + id for permissions (e.g. resource, user, group, etc.)
• original file name
• file reference (specific to storage back end)
• total size in bytes
• format detected or determined from file name
• completion state (ranges received for background/parallel uploads)
• hash(es) (when supported by back end)

Other possibilities:

• name of back end (multiple back end support or for migrating files live)
• support for "files" that are actually links to externally managed resources so we can monitor changes to content based on hash/size when retrieved
• alternate links for redundancy when some services aren't available
• custom fields for permissions, tracking, validation reports or other plugin data

This model would make file metadata reliable, allow us to build new features and potentially save people money by better tracking hosted data in CKAN.

[thegostev]

Hi, @wardi I like very much how the suggestion is described in details. It feels like a change that will be used widely concerning applicability of this change. May you add answers on a standard set of questions:

1. Is this change going to break current installations?
2. Can we provide a backwards compatibility?
3. How easy is gonna be for current implementations to migrate to this new release?
4. Do current versions of CKAN have the adequate resources/support to migrate to this new version?
5. Are we going to change the database schema?
6. Are we going to change the API?
7. Are we going to deprecate Interfaces?

[wardi]

1. Is this change going to break current installations?

No

2. Can we provide a backwards compatibility?

Yes, except for IUploader plugins

3. How easy is gonna be for current implementations to migrate to this new release?

Easy

4. Do current versions of CKAN have the adequate resources/support to migrate to this new version?

Yes, migration will be automatic (except for updating IUploader plugins, see below)

5. Are we going to change the database schema?

Yes, creating one or more new models

6. Are we going to change the API?

Yes, adding APIs for working with files separate from resources/groups

7. Are we going to deprecate Interfaces?

Yes we would need to extend IUploader to support the new features and plugins implementing it would need to be updated.

As an alternative we could implement a new interface and deprecate IUploader instead, to be determined.

[wardi]

updated description above with feedback from dev call: link to #7856 and ownership as part of the model

[mnichol4]

Hey @wardi 👋 Can I ask some clarifying questions?

1. What you're suggesting paradigm whereby one CKAN resource can be associated with many physical files stored in a backend?
2. Where are the benefits of this vs having multiple, linked-by-metadata-somehow filestore resources that have the same data in different formats?

[wardi]

Thanks @mnichol4 these are great questions.

1. Yes, but not by default. This would be a feature an extension could enable in ckan where it makes sense, not an API/UI change that allows attaching multiple files to a single resource for all users.

2. When a site has one primary (user-provided) resource that triggers building content for (system-generated) secondary resources, the ckan API/UI still allows users to edit both types of resource.

   This is confusing and could lead to resources being out-of-sync. Instead we could allow an extension to attach alternate files for different projections/formats of the same primary file then users don't need to be prevented from using the normal API/UI for updating resources and we can guarantee the alternate files will be in sync.

I'd like to see sites benefit from ETL tools like the ones https://open.toronto.ca/catalogue/ has or the summary information generated by tools like qsv #7856, without limiting access to ckan's metadata updating editing features. Auxiliary files managed by a custom workflow extension would be a great use for first-class files in ckan.

[mnichol4]

Ah ok thanks @wardi

So the idea is to have a CKAN extension that allows a single resource to have the option of having multiple files associated with it. That way, each of those files share the same resource metadata.

That's good because it keeps their metadata in sync.
It would also mean that by deleting the resource, all associated files are deleted, so we dont have old, not-touchable-by-ckan files hiding around the backend.

Am I on the right page here?

[wardi]

Yes exactly, no duplicated metadata and everything can be deleted at once

[jqnatividad]

This is much needed for "Metadata resources" (#7856) to work.

Currently, DP+ optionally allows the creation of "secondary" resources (summary statistics, frequency tables, quarantined PII candidate rows) and the connection is a bit brittle. It currently just creates a file with a suffix to indicate the relationship to the primary resource, and a resource extra with the resource_id of the secondary resource.

For the model, you may also want to add "format kind" in addition to "file format", with a parent-child relationship - e.g. "geospatial" format kind - geojson; "geospatial" - kml; compressed - zip; compressed - tar.gz; data - exampledata.json; data - exampledata.xlsx; metadata - datapackage.json; metadata - frequency_table.csv.

Note how the exampledata.json and datapackage.json are differentiated as data and metadata respectively, even though they are both JSON files.

See https://docs.rs/file-format/latest/file_format/enum.Kind.html