ElasticSearch or OpenSearch

[krabina]

@Designburo raised this topic about SMW here https://github.com/orgs/Open-CSP/discussions/5

So the general question is: should SMW stay with ElasticSearch (which since 7.11 has a changed their license model and is currently at version 8.10) or move to Amazon OpenSearch (which is a fork of Elasticsearch 7.10.2.)

For those who are new to the topic, there are some articles for further reading:

• https://en.wikipedia.org/wiki/Elasticsearch and https://en.wikipedia.org/wiki/OpenSearch_(software)
• https://sematext.com/blog/opensearch-vs-elasticsearch-which-one-is-better-sematext/
• https://medium.com/@bhongaleprathamesh/elasticsearch-vs-opensearch-feature-comparison-and-considerations-cd9c990e1482

My personal opinion is that it should stay with whatever decision the Wikimedia Foundation takes with CirrusSearch, but this seems not to be decided yet:

• https://phabricator.wikimedia.org/T272111 and https://phabricator.wikimedia.org/T280482
• https://wikitech.wikimedia.org/wiki/OpenSearch

[krabina]

This will only become valid once we resolve the problem that currently SMW is not compatible with ElasticSearch >7, see #5328

[krabina]

see also #5451

[Designburo]

The problem we face with customers is that Elastic Search 7.11 is already end-of-life and they do not accept that.
Elastic Search itself is a very proven solution and I would favour that. For users that do not use Cirrus Search, it would be fine to use ES 8.x. It is in a way all depending on what the Wikimedia Foundation will decide. However we cannot wait that long any more.

[marijnvanwezel]

We should also consider supporting both ElasticSearch and OpenSearch. I have already started some work on this (https://github.com/WikibaseSolutions/SemanticMediaWiki/tree/elastic-factory-abstraction).

The fact that still no choice has been made indicates that there are good reasons to choose either (e.g. you dislike Amazon, have infrastructure requirements or have a support pact with ElasticSearch and therefore choose ElasticSearch, or you want to use free software and choose OpenSearch).

[krabina]

I totally agree @marijnvanwezel. Thank you for your PR an thank you @JeroenDeDauw for reviewing it!
It is a great idea to support both.

Btw: a question. I think it is a pity that users have to decide between ElasticStore and an RDF store. Do you think, conceptually it would be possible to have the SMWStore to use ElasticStore (which does not replace the SQLStore) or alternatively use ElasticStore AND an RDFStore? This way SMW could provide a SPARQL endpoint and at the same time have the better search experience with Elastic.

[marijnvanwezel]

Hi @krabina, I do not have any experience with how the RDFStore works, but conceptually I think it should be possible. The ElasticStore already uses replication and lives alongside the SQLStore. I don't see why that would not be possible if it were an RDFStore instead.