[Question]: BM25 Retriever takes long time to load with docstore are its parameter

[Prem-Nitin]

Question Validation

• I have searched both the documentation and discord for an answer.

Question

I am using bm25retriever.

BM25Retriever.from_defaults(docstore=get_document_store() , similarity_top_k = 10)

this takes a long time to get initialized. Any way to make it fast.

Another question is, will changing it to MongodbAtlasBM25retriever make it faster

[dosubot]

To address the slow initialization time of BM25Retriever with a document store, consider these optimizations:

1. Adjust similarity_top_k: Lowering this value can reduce computation during retrieval, potentially affecting initialization time.
2. Optimize Tokenizer: Using a simpler or more efficient tokenizer could speed up corpus processing.
3. Corpus Preparation: Optimizing how the corpus is prepared, through more efficient text processing or parallelization, could reduce initialization time.
4. Data Loading Strategy: Efficient data loading or preprocessing can lead to performance improvements.

Switching to MongoDBAtlasBM25Retriever might offer faster initialization if it leverages MongoDB Atlas's infrastructure more efficiently than your current setup. This could include optimizations specific to MongoDB Atlas, such as efficient query execution plans. However, the actual improvement depends on various factors like your MongoDB Atlas configuration, dataset size and complexity, and network connection efficiency.

For detailed implementation and further optimizations, reviewing the base code for both BM25Retriever and MongoDBAtlasBM25Retriever could provide more insights into potential optimizations.

Details

Sources

• llama-index-integrations/retrievers/llama-index-retrievers-mongodb-atlas-bm25-retriever/llama_index/retrievers/mongodb_atlas_bm25_retriever/base.py
• llama-index-integrations/retrievers/llama-index-retrievers-bm25/llama_index/retrievers/bm25/base.py
• llama-index-legacy/llama_index/legacy/retrievers/bm25_retriever.py

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

[logan-markewich]

There's no way to make it faster that I know of. It's loading all the nodes from the docstore that can be slow.

You can get those nodes another way if you want, and pass them in instead of a docstore