Can't support tens or hundreds of billions of data levels？

[hktalent]

It's really great, I don't know
Can't support tens or hundreds of billions of data levels？

[Kerollmops]

Hey @hktalent 👋

What do you mean by data levels? I am not sure to understand.

[hktalent]

I want to turn the hundreds of billions of data I've collected into a search engine

[Kerollmops]

Ok, that's a lot of data 😱

Could you show me the shape of a document?

Only one single instance of Meilisearch will not be able to do the job. However, you could shard your data into multiple Meilisearch by computing the hash of your document id and distribute the database around n Meilisearch instances, i.e. hash(document_id) % n. Then you can use the new Score Details API, that we will release in v1.3, to mix the 20 results you query at the same time from the n servers at the same time, i.e. request_all_servers("my cool query").sort_by(hit["_rankingScore"]).

Have a nice day 🌞

[hktalent]

My current data is currently stored in Java Elasticsearch and Go Bleve, but I have encountered various situations while using them

1. Elasticsearch memory consumption is huge

2. Concurrent massive data writing is slow and does not support distributed multi node collaboration, as well as other situations

The format of the data is JSON, and the members of JSON are not fixed, and the data members may not be the same every time. Requirements

1. Full-text search

2. When JSON member data includes HTML, only non HTML tags are indexed for content

3. I hope to use a certain compression algorithm for the stored data, which means that when different JSONs contain the same data, it is best to use reference relationships internally to reduce redundant storage of massive data

4. I hope to have the support of distributed nodes, as it is difficult for a single node to handle massive data calculations

thanks

[Kerollmops]

Hey @hktalent 👋

With Meilisearch, you'll get a lot of advantages, but it's an engine designed for a search-as-you-type experience and therefore consumes a lot of disk resources to achieve this goal. We are currently working on a replicated data solution by using Raft. What you need is sharding, which we will not release soon.

Would it be possible to have more info on the size of the documents and shape?
How many fields to search on and filter on? How many words on average?

However, if you are ok with losing some features like typo tolerance and the search-as-you-type experience you should try Quickwit, which looks much more adapted to your needs. You can store the documents in an S3 file system and feed it with thousands of billions of entries, and Elastic Search inspires the API.

Have a nice day 💾

[fulmicoton]

@hktalent Hello! I'm the CEO of Quickwit. Thanks @Kerollmops for mentionning us.
Quickwit is designed for this volume of data. We allow partial schemas too.

   1. We do Full-text search. That part is very close to Elasticsearch.

　　2. When JSON member data includes HTML, only non HTML tags are indexed for content
We don't have any tokenizer that supports HTML yes, but this should be reasonably easy to add.
Within quickwit or outside of quickwit, before you input the data.
3. We use ZSTD on rather large blocks
4. Quickwit is distributed. We handle up to 100 nodes.

If you want to continue this conversation the best is to come on our Discord, or on our github.

[hktalent]

@fulmicoton @Kerollmops Thank you very much. I will consider it again

[ip-rw]

This was a helpful issue and saved precious internet by avoiding the need for me to ask!