CKAN future as we want it to be

[thegostev]

Hi, CKAN community fellows.

It looks like a good topic for an open discussion: what we want CKAN to become in the future. What things we want to add, what’s missing? What are current problems that we can transform into objectives, to solve them?

With CKAN 3.0 we received commitment from community members that made it possible to have a scope for this major update:

• Modernizing tech stack with HTMX addition (by @pdelboca ).
  Pilot integration delivered, more inclusions to come.
• Ability to integrate ElasticSearch as an alternative search engine (by @dragan).
  70% done at the moment. The delivery will happen in December 2023.
• UI Revamp (sponsored by Link Digital and Datopian, contributors are yet to be selected).
  Team of 2 should be formed at the start of December 2023 with preliminary estimate of 4 months to completion.

The importance of this collaboration is in forming a vision for CKAN in 2024. After CKAN 3.0 is delivered we have to have a direction where to move and it’s a perfect moment for us all to engage and think it through beforehand.

[thegostev]

My 5 cents 🪙 is that we should strengthen the operational side - to concentrate on getting commitment from willing contributors while at the same time looking for sponsors from governments and non-profits. I believe if we’d find organizations that would believe that we’re doing a great job by moving the open data forward, we can get necessary support and make the future happen.

[jqnatividad]

At datHere, we would like to nominate for consideration:

• Data Resource Upload First (DRUF) as part of UI/UX revamp
  Elements of this is already done with Datapusher+ and the work on Scheming v3.0.
  DRUF was informed by years-long User Research by TWDB that specifically spoke to the UX limitations of CKAN. TWDB has indicated that they're willing to share their User Research findings with the project. IMHO, this can be included as part of the UI revamp, as UX drives UI.
• DCAT 3.0
  IMHO, there are certain DCAT 3.0 novel concepts like versioning and making geospatial data first-class concepts in CKAN that should be investigated for inclusion into the CKAN 3.0 roadmap. DCAT 3.0 has been officially published Mar 2023 and the two popular DCAT profiles - DCAT-AP and DCAT-US will also be ready by 2024. DCAT-AP was published July 2023, and DCAT-US is targeted for December 2023.
  As governments account for a large percentage of CKAN's market, this could also be an opportunity to convene Communities of Practice/Working Groups to directly engage key stakeholders in shaping CKAN 3.0's support of the DCAT 3.0 specification - which if I'm not mistaken, will eventually become mandates in the not so distant future. This conveniently dovetails into the POSE Phase II proposals.
  Fortunately, we are also collaborating on a small pilot with a US federal agency that will implement aspects of DCAT 3.0, and we can contribute that to the project.

[jqnatividad]

CKAN 3.0 also presents an opportunity to upgrade and modernize critical subsystems - as is being done with HTMX and alternative search engine support.

We should also consider:

1. bumping minimum PostgreSQL version supported to the the latest PostgreSQL major version when CKAN 3.0 development starts in earnest.
2. Upgrading to SQLAlchemy 2
3. Using the latest stable version of Python 3 when CKAN 3.0 development starts in earnest
4. Preemptively use the next Ubuntu LTS release - 24.04 as baseline OS. By default, it will have PostgreSQL 15 and Python 3.11 (see 1 and 3).

So long as there is a migration path from CKAN 2.x to CKAN 3.0 (which could just be a database migration, not an in-place migration) I think 3.0 is a big opportunity to do these breaking changes that allow us to retire a lot of technical debt, get "free" performance improvements, while tapping the latest features (e.g. SQL/JSON path expression support introduced in PostgreSQL v12, etc.) afforded by the latest versions of these underlying subsystems.

[anuveyatsu]

@jqnatividad thanks for this - I agree these could be important to ensure consistent and stable performance of the software. Also makes it easier to test/upgrade/release new version etc.

[anuveyatsu]

DCAT 3

Most of the points above make sense to me - I'd only highlight that with DCAT 3 we might want to make changes in the default metadata schema which I think a little bit outdated. I believe that with DCAT extension providing DCAT 3 support could be relatively minor work but major feature for data publishers.

We already have this issue #7896 by the way 😄 cc @thegostev

ETL

As it was initiated in #7492 I would suggest adding a tutorial/docs/instructions about how an external ETL system can be integrated. Having a separate page in https://docs.ckan.org/en/2.10/ would be amazing. We can use Airflow integration via Aircan to provide predefined DAGs that can be used out-of-the-box or extended based on each project needs.

[popovayoana]

@thegostev who can take care of this? can we create an issue for it and push it forward? i can assist if needed with polishing/coordinating etc // cc @anuveyatsu

adding a tutorial/docs/instructions about how an external ETL system can be integrated (having a separate page at https://docs.ckan.org/en/2.10/)

[thegostev]

Hi @popovayoana we need content first, then adding it to docs is a straightforward part. If you know anyone in Datopian who can write the instructions it may may be the way.