Road Map - Elastic Search and Open Search

[DreamDelerium]

Hello. Is there any planned support for Elastic Search and/or Open Search? If so, what sort of time table is expected for that? Thanks!

[pdelboca]

Hello @DreamDelerium ! There is a strong desire to support Elastic Search however currently there is no capacity to deliver it.

As a community we all agree that CKAN shouldn't be so tightly coupled to Solr and that a middleware to be able to choose from different full text search solutions will be a great addition.

CKAN 2.10 (that will be launched in the upcoming weeks) will address most of the technical debt that the project has been struggling with for years leaving the codebase (and the backlog) in a good state for features like this one. Supporting of Elastic search is one of the most requested features so I guess that it will be in the roadmap for next releases. (Example of other people wanting ElasticSearch: GSA/data.gov#3779)

There's a lot going on right now so providing a timetable is almost useless and will create false expectations, but for sure I can say it is not something that's gonna happen in the next months.

I would like to repeat that there is an agreement that CKAN should support ElasticSearch so any external innitiative or PR to address this will be highly appreciated, supported and prioritized by the Tech Team.

[DreamDelerium]

Thank you! I appreciate the thorough response! Looking forward to seeing the new version!

[jqnatividad]

As we think about search in light of CKAN 3.0, we should take the opportunity to revisit CKAN search in general:

• datastore record-level search, and what's the interplay between PostgreSQL full-text-search and the Catalog metadata search
• indexing non-tabular resources (PDFs, Office documents, etc.)
• searching the data dictionary
• spatial search
• searching the Activity Stream?
• recalibrating the CKAN user experience with user's search expectations in 2023 (typo tolerance, suggestions, predictive search, search as you type, search highlighting, semantic search, etc.)_
• how custom scheming schemas are indexed, what is considered searchable metadata and if the schema developer can indicate if a field is searchable and what its search weight is
• be open to considering other open source search engines (like Meilisearch, which is ludicrously fast whilst being simpler to deploy/maintain)

[jqnatividad]

With Datapusher+, there are also additional search-related considerations:

• fields in the Datastore are now always stored using their native data type, and not just text
• now that recline_view is going away and DataTables_view is the default, can we retire the existing filter function and use DataTables' SearchBuilder plugin instead, which is a far more sophisticated, data type-sensitive query builder?
• should we continue creating full-text-search indices by default when creating datastore tables?
• shouldn't we make the data dictionary part of the search index?
• DP+ compiles summary stats very quickly, even for very large datasets using multithreaded, streaming (constant memory) scans (e.g. NYC's complete 311 archive from 2010 to 2022-03-04 - a 27.8 m rows, 41 columns, 16gb CSV takes only 25 seconds to run on my 16gb M1 Mac Mini consuming only 9 mb of memory while using all 8 cores). Can we not add these stats as "system-level" metadata? For example:
  • automatically setting the date range of a dataset based on the min/max values of a date field that was annotated as indexable in the schema?
  • automatically setting the spatial extent of a dataset by using the min/max values of the lat/long field
  • adding the top N values of each field into the index so that you can still effectively search at the record level without having to index the entire dataset