Decouple frontend to simplify onboarding for frontend developers

[anuveyatsu]

I'd like to see what people think about decoupling the frontend application for the following reasons:

• Onboarding is simple, e.g., you just run 1-2 npm commands (npm i && npm run dev) to get started with CKAN APIs mocked.
• It attracts a lot of frontend developers as it would be easier to start contributing. Developers wouldn't need to know Docker, Python etc. It would be a common tech stack for frontend engineers.
• More frontend contributors would mean our community would extend and our UI/UX would improve.

[pdelboca]

Hello @anuveyatsu ! How are you?

How do you envision a decoupling frontend for CKAN and what are the next steps you suggest for it?

Current CKAN Architecture can be defined as a 3-Tier architecture and it provides a rich logic layer and API to access all the important business logic. This means that in theory, regardless of CKAN shipping with a built-in front end other decoupled systems can be designed and developed. CKAN API is well documented and quite stable so they can be mocked to work on a decoupled front end.

Now, I'm conscious that our layers are not truly independent and there are some business in the model and model in the business that might become pain points for other decoupled frontends so that's why it would be nice if you can expand your initial thoughts.

I know that there are some current efforts to build a decoupled front end like https://github.com/datopian/portal.js so I'm wondering what are your thoughts on what do you envision in the practice as a decoupled CKAN. Would you like to do a presentation in the Dev Meeting with all the work you are doing so we can explore better your ideas?

Another question that pops in my mind: What thing of the current CKAN implementation does not allow to build a decoupled front-end? If someone wants to build a decoupled one, what can we do different to support that effort?

Looking forward to keep the discussion!

[anuveyatsu]

Hey @pdelboca thanks for the detailed feedback 🙏🏼

Another question that pops in my mind: What thing of the current CKAN implementation does not allow to build a decoupled front-end? If someone wants to build a decoupled one, what can we do different to support that effort?

I think I'd start with this question - nothing stops people building decoupled frontends. Probably, what hesitates teams to do so is the fact that CKAN is shipped with frontend built-in which means if there are changes in frontend (e.g., new features added or bugs fixed), they wouldn't get it via upgrade but would need to make extra effort to re-build it themselves etc. To summarize, it creates uncertainty about how it is going to be managed in the future.

I agree that in theory anyone could use their preferred framework and build their desired frontends that works for their team's tech stack. We've seen number of examples using React, Next, Vue, Wordpress etc. I believe some of them have built great UI/UX for both publishers and consumers of the datasets. But I still think that major part of developers building on CKAN had the following experiences (this is based on my personal experience working with different clients / our own developers at Datopian etc.):

• Did frontend using default approach (jinja2, bootstrap 3.3.7 (was upgraded recently to v5)), however, it wasn't their preferred choice.
• Built React/Angular/Vue/etc. apps and then packaged into CKAN extension which created more complex maintenance workflow.
• Struggled with having local dev environment for just doing light branding of the user interface.

How do you envision a decoupling frontend for CKAN and what are the next steps you suggest for it?

This needs discussion but my first thoughts would be:

• Have a separate API app which would purely provide with APIs, e.g., you setup current python app and get everything except views layer.
• Have a separate app which will be built purely frontend stack and provides UI/UX based on CKAN APIs.

Roughly:

• github/ckan/ckan
• github/ckan/frontend

The frontend app would use something common/popular today. I think we could have it as a test for next steps and see what would be number of contributors from the community, e.g., those who never did a contribution to the ckan/ckan. If we get attention and good number of contributors, we could say this is a success - we have X number of new people in Y months, we have shipped ABC etc.

To sum up, I think we could simply start a new frontend repo with some prerequisites, make an announcement and track what happens. This way the core code remains the same, we don't do any changes yet.

Would you like to do a presentation in the Dev Meeting with all the work you are doing so we can explore better your ideas?

Yes, I wouldn't mind doing so, if people are interested in exploring more in this direction.

[pdelboca]

Interesting thoughts @anuveyatsu and definitely worth exploring for upcoming CKAN versions. Some follow up thoughts...

Given two repositories ckan and frontend:

• What would happen with extensions that needs to add UI as part of their functionality? Would they also need to create two repositories?
• How would a new extensions (like ckanext-pages) frontend interact with the main CKAN frontend app?
• What would CKAN implementators need to do to customize the front-end?

Basically, current workflow of CKAN work is the following:

• Install CKAN
• Create a new extension for styling + custom business logic
• Add a couple of existing extensions to their stack.

One of the key features of CKAN is the plugin and extensions ecosystem. How do you envision the interaction of plugins ecosystem with a new decoupled schema?

[pdelboca]

I think that a quick MVP pilot to simulate how CKAN workflow would be under the new decoupled schema is key to analyze the viability of the solution.

I'm thinking on simulating a scenario that:

• Installs CKAN's backend
• Installs the new front-end
• Customizes the frontend to adapt logo and basic style (like any user of CKAN does)
• Creates a new extension extension that can be plugged in/out to provide a new feature.

It doesn't need to be fancy, just to provide an insight on how the architecture and the interaction between plugins would be.

Do you think the current ecosystem of decoupled frontends is mature enough to do it? If not, do you think you can provide a quick pilot simulation?

[anuveyatsu]

Again, great comments 👍🏼 @pdelboca appreciate these questions and you raised important points from the maintenance and development perspectives.

First, note that in the frontend app, there might be different approach for extending, not exactly how we do it for CKAN app.

Generally, what we are discussing here is making CKAN a headless tool. We can compare this against how CMS softwares are approaching it. For example, a headless CMS's main goal is to provide content and expose it via API so that you can use any frontend technology. In CKAN, we could take similar approach which would mean publisher and consumer (regular user) UIs are separated (more about it below).

In CKAN, we have 2 main groups of users - Data Publishers and Users:

• Publishers can manage datasets.
• Users can read public datasets + follow certain entities (dataset, group, org). With some extensions they can also provide comments etc.

So the main questions to me:

• Do we want to have both of these in the single decoupled frontend app?
• Or do we want to have 2 separate UIs for these 2 groups of users?
  • Then, do we want to have Publisher UI also separated/decoupled or does it remain part of the python app?

Option A: both UIs in the single decoupled frontend app

This option would mean we would need to have all UI in the single app including the extensions, e.g., ckanext-pages UI would be there. If developing a new extension that adds both backend changes and frontend changes, you'd have to develop in the both apps which is fine I think. The main issue here is that the frontend app can get complex by adding a lot of logic and I think it should remain minimal.

Option B: UIs are separated, publisher UI remains in the python app

This is the easiest and quickest way to go right now. E.g., the main CKAN app doesn't change and we simply create frontends (e.g., templates and it can be in multiple popular frameworks). Users then can build on top of these templates.

Option C: UIs are separated, publisher UI is also decoupled

This option probably is the next step following Option B. Since user UI is decoupled, there is no need to have home page, search page etc in the Publisher UI and it can be focused on things like inventory, dataset management, user management etc. Could be another decoupled app which would focus on specific thing - Publisher or Admin UX.

[pdelboca]

Helllo @anuveyatsu !

One thing that worries me about the decoupled architecture is the extensibility and re-usability of the tools. CKAN is not a data portal, but a framework to build ones. CKAN is to Data Portals what Flask/Django are for websites.

Said that, The goal is to provide a core that allows other partners to build on it and share. IMHO, the modular and extensibility is a key feature of CKAN that needs to be maintained.

Example

Let's look at ckanext-spatial which adds geospatial capabilities to CKAN. The extension:

• Adds new fields to the database
• Adds new search capabilities into solr
• Adds new snippets to the UI (both for publishers and consumers)

How would you approach a development of such extension in a decoupled ecosystem that also allows a "plug-and-play" into other implementations?
Are we going to have backend plugins that provides the data and API and front end plugins that provides the React/Vue components?

[wardi]

The suggestion is to replace jinja2 with "React/Angular/Vue/etc." and @pdelboca has highlighted the most important questions about this change.

Jinja2 is not perfect. Our jinja2 templates create a very large interface that is difficult to change without breaking backwards compatibility, and our custom tags like {% ckan_extends %} and {% snippet %} add confusion and have performance drawbacks.

However, jinja2 template have proven themselves in allowing multiple plugins to extend all parts of the UI, along with complete site theme changes. They've done this while allowing broad compatibility across plugin and ckan versions too.

Jinja2 intentionally has limited "programming" features to encourage writing mostly plain HTML so it's easy to understand, extend or override. Is the same approach available in React/Angular/Vue or is JS code freely mixed with bits of HTML requiring a JS developer for any change?

---

Having separate repos for ckan and its frontend would mean tightly linking the versions of those repos. Any change affecting both would need PRs on both repos that would need to be considered together and merged near-simultaneously. That makes tracking issues with git bisect more challenging, and generally creates more busywork for core developers.

The benefit to outweigh this new friction is to "simplify onboarding for frontend developers" i.e. JS developers. This assumes we have lots of people with JS programming backgrounds that want to contribute to ckan, but the opposite has been true.

---

I would like to see better APIs to enable responsive front ends for ckan. This could be something like graphql, new endpoints that return all the information required for rendering a page in a single call, or something like htmx returning bits of rendered HTML to replace parts of pages on the fly.

Any of these would enable core front end improvements and custom front ends, allowing those with JS expertise to build the amazing targeted front ends they like, while core UI remains comprehensive.

[jqnatividad]

Hi @anuveyatsu!

We're currently involved with a project where they have a hybrid setup - where they built a React-based CKAN frontend loosely coupled using ckanext-sso with "CKAN Classic"'s built-in UI, themed using conventional jinja2 templating techniques to look like their custom front-end, where they implemented their unique custom data ingestion workflows.

Custom data ingestion workflows like PII screening; super-secure S3 storage instead of using CKAN's File Store via ckanext-cloudstorage; loading and downloading resources from the cloud - i.e. dropbox, sharepoint, etc.

These workflows called for a custom UI/UX with all kinds of business rules implemented using Lambda and Javascript and UI controls that was ill-suited for scheming, or even a custom IDatasetForm implementation.

For the most part, it works, and they were able to delegate things like the solr-powered faceted search interface to the themed "CKAN Classic" built-in UI, and were able to use several CKAN extensions as is.

Even though they implemented their data ingestion/entry using their custom front-end, they still used scheming and took advantage of its ability to quickly customize the package/resource metadata schema, and primarily used it for editing metadata, as they liked how they can use the classic UI for viewing the Activity Stream. They actually quite liked scheming and how it made extending/customizing the metadata schema much easier, but they needed more flexibility with the data ingestion process.

Another client even supported the development of form-pages in scheming 3.0 that @wardi recently released, because they also liked how scheming made it easy to customize metadata - non-trivial capabilities that will be hard to implement on a decoupled front-end.

As @pdelboca pointed out, "CKAN is to Data Portals what Flask/Django is for websites."

Perhaps, we should embrace this and even encourage it!

Embrace this "hybrid" approach showcasing CKAN as a framework - where core stuff that's hard and non-trivial like faceted search; editing datasets with customizable metadata schema replete with form pages and validators; having the ability for plugins like ckanext-spatial extend the UI/DB/API remains in the CKAN DMS framework.

So having the current scheme becomes a feature of the framework, not a bug.

Perhaps, instead of decoupling the front-end, we instead create a repo where we embrace and enable front-end developers who wish to create custom front-ends with several reference implementations using popular front-end technologies/frameworks?

My 2 cents,
Joel

[anuveyatsu]

Hi @pdelboca @wardi @jqnatividad you've shared great points and I totally understand your concerns. If I can summarize this is what we're discussing:

• CKAN is a framework that's easy to extend and customize and it should remain so.
• CKAN's frontend tech stack could be changed if it brings improvements without additional complexity in the maintenance.

If we could find an approach that could satisfy both of these points, I think we could move further.

Let me know if that's true so I can think about proposals here!

[pdelboca]

Hi @anuveyatsu ! Yes, that's kinda what are we discussing. I could rephrase your points like:

• CKAN is a framework that's easy to extend and customize and it should remain so.
• CKAN is a framework and should provide a toolkit to develop reusable extensions and plugins. (With an emphasis on "plug-in")
• CKAN's frontend tech stack could be changed extended if it brings improvements without additional complexity in the maintenance empowers 1 & 2.

The same way that people who wants login in their Flask application can pip install flask-login, users who wants better authorization in their CKAN should be able to pip install ckanext-saml2auth. - Lao Tse, Tao te ching.

I prefer using extend frontend instead of change. On the ground, saying "we are going to change our frontend from X to Y" will trigger the question:

• "What happens with all the features that we already provide for X?"
• "What happens with all the current implementations based on X?"
• "Can I still use an extension I did for X but with Y?"
• "Is ckanext-schemming gonna be supported on Y?"
• "Are we going to keep providing support for X or drop it?"
• etc, etc, etc

Also, if CKAN is really decoupled, then having a default Jinja solution shouldn't affect the development of other frontends so I don't see any harm on keeping it. One could argue that it's more maintenance, but the idea of decoupling also implies that multiple frontends can be developed with independent communities.

If "decoupling" means that we should change our current template system that means that we are decoupling it wrong. That's why my initial question was: What thing of the current CKAN implementation does not allow to build a decoupled front-end? To which you answer:

Probably, what hesitates teams to do so is the fact that CKAN is shipped with frontend built-in which means if there are changes in frontend (e.g., new features added or bugs fixed), they wouldn't get it via upgrade but would need to make extra effort to re-build it themselves etc. To summarize, it creates uncertainty about how it is going to be managed in the future.

But that doesn't make too much sense for a couple of reasons:

• There is plenty of certainty of who things are gonna be managed in the future. CKAN is a stable platform that provides good support for backwards compatibility. We put a lot of effort on it so new upgrades do not break current installs.
• Saying that people don't want to implement a decoupled solution because they would need to make extra effort to re-build it themselves beats the purpose of decoupling. It's like saying that people don't want to use GNOME because KDE implements better features.

So, long story short:

Your proposals are welcome @anuveyatsu and I know you have lots of experience with decoupling CKAN, so as long as we keep it extensible and reusable we can move forward on improving the ecosystem! :)

**Note: ** I'm not the official voice of the CKAN community so on each of my points reader should prepend a: IMO.

[thegostev]

Hi, everybody. It's already quite an insightful post. @anuveyatsu thank you for starting the conversation.

CKAN is not a data portal, but a framework to build ones. CKAN is to Data Portals what Flask/Django are for websites.

A brilliant explanation 👏 @pdelboca

I would like to see better APIs to enable responsive front ends for ckan. This could be something like graphql, new endpoints that return all the information required for rendering a page in a single call, or something like htmx returning bits of rendered HTML to replace parts of pages on the fly.

Any of these would enable core front end improvements and custom front ends, allowing those with JS expertise to build the amazing targeted front ends they like, while core UI remains comprehensive.

I like this idea. It does lead to anticipated improvement with less risk. It's not a radical though.

Perhaps, instead of decoupling the front-end, we instead create a repo where we embrace and enable front-end developers who wish to create custom front-ends with several reference implementations using popular front-end technologies/frameworks?

Sounds like a nice thing to do, to capture work that can be reusable. The downside is that we don't have that many frontenders to contribute, for now.

Your proposals are welcome @anuveyatsu and I know you have lots of experience with decoupling CKAN, so as long as we keep it extensible and reusable we can move forward on improving the ecosystem! :)

@anuveyatsu do you have an idea of how to de-risk the change so that we're moving to simplification of developer experience?

Cool topic 👍 Thanks,
Alex.

[anuveyatsu]

@pdelboca @thegostev thanks for more comments!

Initial proposal for having a separate repo was for sort of PoC and if everyone is happy about it only then we would consider getting this into main branch of CKAN. Having it there would mean we could consider options such as monorepo to reduce maintenance/integration testing efforts. If people like the idea of having frontend(s) in the ckan/ckan as npm packages or similar, we could explore that path. Generally, I believe this is only the beginning and it requires thinking and writing down analysis/research rather than actual development. I think the development is relatively easy once everyone is onboarded.

@pdelboca regarding backwards compatibility, CKAN is using semantic versioning and with major version release they should expect some incompatible changes 🤔 but obviously it must be straightforward to upgrade with a clean instructions. Nevertheless, I think this discussion is out of scope for now.

[amercader]

I'm late to the discussion so I don't want to add much noise to it, but here are some of my thoughts.

First of all, I believe all these statements are True at the same time, and probably pretty uncontroversial:

• Server-side rendered templates like Jinja2 integrate seamlessly with the existing CKAN authorization, blueprints mechanism and plugin system, in a way that has been tested over many CKAN versions / years.
• The current Jinja2 based templates could do with an improvement in all fronts: visually, performance-wise, structure of files / blocks and in terms of the javascript being used
• CKAN needs a better onboarding experience for new developers, specially those not familiar with Python-based backend development
• JS based applications based on frameworks like React / Next / Vue / etc are very popular¹ and there's a large pool of developers familiar with these technologies (almost certainly bigger than those familiar with Python web development)
• The technologies, ecosystems, tooling, patterns etc involved in developing these frameworks are different from the ones involved in developing a Python web application. It's not common to find developers with ample experience in both
• The current Tech Team skills lean more towards the Python backend side of things (nobody get mad now, I know you know JS!)
• One of the biggest success factors of CKAN has been its extension system and the ability to customize its behavior to accommodate custom needs. Whatever frontend is used should allow that
• There are limited resources available for core CKAN development, and we are stretched thin

The following I also believe to be True, but are more based on my personal opinion and experience with the CKAN community:

• If we set aside big Vendors that maintain and develop several CKAN instances and have big multi-skilled teams, most of the rest of CKAN sites are maintained by small teams or even individuals ²
• The task of customizing the frontend for a particular instance, while critical, is just one of many common ones when customizing a CKAN site, alongside customizing the schema and validation rules, add custom business logic, integrating with authentication systems, etc

And the following are purely personal opinions (neither true or false :) )

• We are not far from booting up a working CKAN instance / backend in a few commands. Our Docker setups or even cloud based envs like Codespaces are not far from that, if they not already allow it
• Any frontend dev having to run npm i && npm run dev will also have to setup the backend somehow (again, I think the case of frontend devs having to work on site already set up by someone else is quite limited)
• Nothing stops us to integrate rich reactive JS UIs in Jinja2 templates for an hybrid approach that would greatly simplify the setup
• The CKAN API should support the development of a completely decoupled frontend

Based on all these, here's what I think could be a good approach

• I just don't think is viable to completely replace Jinja2 with a decoupled frontend in one go, so we make clear that we'll keep the Jinja2 templates even if work on a decoupled frontend starts
• We should start a process of discussion around next steps for the Jinja2 templates. CKAN 3.0 means that we can break things, and be bold about it
• I think it would be super-valuable to have a default implementation of a decoupled frontend in a separate repo that people can plug into their sites to try it out, and identify what works well and what shortcomings are there. We can definitely advertise it and ask for testing. @anuveyatsu given your experience with portal.js I think Datopian is perfectly placed to lead on that, with contributions from the whole community
• If the decoupled option gets traction, we come up with proper integration tests that ensure it's stable for a particular CKAN version and turns out to be a viable option for users I'd be in favour of offering both options in the install instructions so users can choose what works best in their particular circumstances

Sorry that ended up being more noise than I expected but hopefully there's some path forward there

Footnotes

1. Although it's fair to say that there is a significant movement of influential people raising concerns about client based apps and SPAs shortcomings and about them being overused ↩

2. Maybe it's time for another big developer survey to have a better picture of this ↩

[pdelboca]

Gonna close this discussion since it seems that a kind of agreement has been made around using HTMX for the new frontend: #7524