New extensions portal: CKAN-based, dogfooding, could grow into CKAN-wiki

[thegostev]

It's the idea to replace the current extensions portal with a CKAN instance that we'll be maintaining. It's a chance for us to increase plugins re-usability with a sizable additional benefit to start using own product (it'll lead to formulating requirements for UX update for publishers).

• What do you think about this idea?
• What the current portal is missing that we need to have in a new one?
• What are good features we can borrow from other marketplaces/app catalogs?

Background for the project

Current extensions website isn’t maintained and reached a dead-end. Concerning that plugins is one of the most valuable parts of CKAN ecosystem it’s important to make them findable, well-described and the website easy to use. It should have a feel of app marketplace for better accessibility.

We want to use own product (CKAN) to catalogue extensions. Next stage would be around adding other elements of the ecosystem to enable web of data and cross-referencing:

• Extensions are used by these portals
• These are portals for cities with population over 1 mln.
• These are plugins that are usually used with plugins you have

Users could get notifications on what we can suggest to them.

Problem formulation

Problem(s)

Plugins cut off development time and extend the system in no-time. Getting plugins used increases value of the ecosystem. Current extensions website neither convenient nor helpful enough.

Value

• Increase in plugins re-usability: Reused code = work of plugin creator matter. The more code is reused, the less it’s a need for writing new code. Thus it’s easier to create implementations.
• Increase in plugins discoverability: value of the CKAN as a system grows when an implementor is able to find a suitable extensions for the system.

Next version

• Notifications for new versions of plugins (plugin <> catalogue), security

Desired outcomes

• Increased plugins re-usability: Amount of plugins used ⬆️
• Increased plugins discoverability: Website traffic ⬆️

Next version

• Notifications: Amount of returned users ⬆️ MAU ⬆️

Hypotheses

• By having a clear structure in catalogue it’ll be easier for people to discover plugins
• By making it visible which plugins are featured, created by trusted creators, updated recently - it’s possible to find a plugin that will most likely work fine and have a good quality
• By providing an ease of submission for a plugin we can make more up-to-date plugins listed
• By interlinking data between plugins, instances, events, organizations, people it’s possible to create a deep experience and higher amount of time on the portal
• By notifying CKAN website maintainers about issues with their instances we can engage them to return regularly to the extensions portal

Questions/Existing research or customer insight

Can we research what will give us:

• Adding social component to have discussions around extensions

• Interlinking

• How we’ll do interlinking organically (without massive development) work?

• Who will do the data migration between the current extensions website and a new portal?

• Can we predict and advise what extensions user needs based on his parameters (type of business, planned amount of datasets, domain, etc.)?

User insight

Insights from CKAN research

• Officially featured extensions
  • It’s a complimentary for a self-service component
  • It’s a good incentive to contribute to the ecosystem
    • A lot of people take but never give back, we should incentivize to contribute
• Guide on how to polish own custom extension for it to be endorsed
• Plugin version certifications
  • Guaranteed to work with 2.10, 2.09
  • Tested on 2.10, 2.09

Timeline for iterations/activities

1. CKAN instance launched at [extensions.ckan.org](http://extensions.ckan.org) with necessary updates to support plugins handling
2. Most important plugins (20 at least) moved to new platform, old website is moved to oldextensions.ckan.org
3. Add “submit a plugin” functionality with moderation option
4. Apply CKAN styling
5. Add ability to add people, organizations
6. Add ability to add events, case studies

[pdelboca]

+1 on having a better extension visibility.

On the one hand, a whole new site with social features might seem a little bit too much in terms of efforts and current capacity. On the other hand, having a customized CKAN instance for this could be beneficial if we also use it to test integration for new releases.

A smaller first step could be to update the current Extension Monitor so it can work as a catalog too: https://github.com/ckan/ckan-extensions-monitor.

I envision that repository with:

• An extra column for the repository small description
• An extra column for tests passing in the last CKAN version (not only master branch)
• Different sections for community driven extensions and vendors extensions.

It is a simple approach and has several benefits:

• It would be easy for vendors to showcase their plugins. Built in "submit a plugin" functionality (A small PR is enough).
• It would give in a glimpse the sanity of the ecosystem (How well the community/vendors maintains their plugins).
• It's on Github, which is the natural place of CKAN ecosystem of plugins.
• Doesn't require too much maintenance.
• Doesn't require custom development.
• Engagement is easy.
• The repository already have a forum for discussion/debate/Q&A.
• It's already up and running :)

On the downside:

• It provide limited features and showcase (it's mainly focused on extensions)
• We do not have events nor case studies.

[thegostev]

Thank you. It could be interesting to integrate extensions monitor with extensions portal to cover compatibility. As we'd be using CKAN to power the portal I don't expect too much of development work. Migration will take time, discussing with @jqnatividad creating script for migrating data.

That's also great that you've mentioned that test integration for new releases is a side benefit for having own CKAN instance. For me it's a valuable additional benefit of dogfooding.

Regarding listing plugins in GitHub, I like ease of contributing and minimal maintenance costs.

[tino097]

Not so long ago we created this repo.

[tino097]

hey @thegostev i think only @rufuspollock has privileges for that

[rufuspollock]

I'd +1 awesome-ckan as a lightweight way to get started.

BTW: Intention was for awesome-ckan to be wiki-like and certainly manageable by ckan org - I wasn't aware others couldn't edit. Currently it says all members have access plus "awesome-team" (which is only 3 people atm: myself, amercader and zoran):

I've added core team.

[thegostev]

Thank you @rufuspollock I can edit the file.

[thegostev]

I really love simplicity of keeping references in GitHub. After expanding the extensions table with monitoring badges (@pdelboca would it be possible to add them to the Awesome list?) the list will have an additional advantage of tracking extensions.

I don't really treat the Awesome list as a competing solution, rather like an additional channel to spread the word about existing extensions. The list in GitHub is cool as it solves the problem cheaply and simply. It should be interesting to expand on both ways (if we'd be able to fill the CKAN-based portal and maintain it) and see what will stick with users.

[anuveyatsu]

Added this issue which relates to this discussion - ckan/ckan-instances#121

[jqnatividad]

Thanks @thegostev for this!

IMHO, a CKAN-based extensions portal is a great idea as it accomplishes several things.

• the current extension catalog is a metadata catalog about extensions, we're not pushing a circle into a square hole.
• core CKAN concepts like organizations, groups, packages & resources map to what we need to track about extensions
  • users - the submitter/maintainer/user of the extension
  • organizations - for attribution/ownership and ability to do self-service entry/editing of metadata
  • groups - for grouping packages under certain themes (e.g. visualization, authentication, etc/)
  • packages - for the extension metadata itself - to track versions, which CKAN it is "certified" to work with, freeform tags, etc.
  • resources - to link to various resources about the extension like the github/gitlab repo, a presentation about the resource, etc.
• we can use existing extensions to flesh out certain requirements.
  • scheming - to create the extension, case study, and CKAN instance registry schemas
  • showcase - to opt-in catalog CKAN instances that use the extension
  • datarequests - for requesting extensions
  • disqus/discourse extensions - to have conversations, etc. Or to @pdelboca's point, maybe we can use https://giscus.app/ to use GitHub discussions on the extensions repo instead of disqus/discourse

We actually built a CKAN-based portal six years ago. I was maintaining extensions.ckan.org and was keenly aware of its limitations from the backend, and as a member of the CKAN Steering Group then, I advocated for it and actually went ahead to build a prototype - https://github.com/OpenGov-OpenData/ckanext-extensionstheme.

We even created a script to use the existing catalog to seed the new instance.

[thegostev]

Thanks for your contribution @jqnatividad. It reads that CKAN has a natural out-of-the box capability to serve as an extensions portal. Combining it with discussions from GitHub looks like a win-win. I'm so thrilled to get a living CKAN instance that the CKAN-based portal looks attractive. Aside of sizable additional benefits it's a great fit as you broken it down.

I guess that most of concerns are around proper maintenance of it and how we'd fill it out with content. I guess if these 2 parts of the job are solved, we can try this way. I believe it's possible to launch an instance and see if we'd be efficient enough to fill it out and become a replacement to the current portal. If this line of work will be failed than minimalistic solution of keeping list of extensions in GitHub is what we can afford (despite our appetite 😋).

[jqnatividad]

Folks,
The Pittsbugh POSE Team compiled a survey of CKAN instances as part of our Phase I Ecosystem Discovery:

https://civicdataecosystem.org/2023/06/19/analysis.html

I refer to it here as I commented there that I still believe a CKAN-based Extensions Catalog/"Ecosystem Catalog" is better suited to the job than a manually curated Wiki.

cc @thegostev @rossreillypitt