[Bug]: BM25 Retriever - Corpus uses default MetadataMode while reading content from nodes instead of MetadataMode.EMBED or user provided option

[gopalgk]

Bug Description

In BM25 retriever the corpus is built from nodes using the default argument to get_content() as shown below.

self._corpus = [self._tokenizer(node.get_content()) for node in self._nodes]

But, if the nodes have a specific metadata to include/exclude while searching then we cannot enable it. When we build hybrid retriever (using vector index and bm25 retrievers) the content that gets passed to vector retriever will be different than bm25 retriever. Since it is expected for retrieval we should use MetadataMode.EMBED then the same has to be passed for bm25 corpus generation as well (below).

self._corpus = [self._tokenizer(node.get_content(metadata_mode=MetadataMode.EMBED)) for node in self._nodes]

Version

v0.10.35

Steps to Reproduce

Use this notebook and add metadata fields to the nodes. Set excluded_embed_metadata_keys in the nodes to []. Embedded search will use different content than bm25 search.

https://colab.research.google.com/github/jerryjliu/llama_index/blob/main/docs/docs/examples/retrievers/bm25_retriever.ipynb

Relevant Logs/Tracbacks

No response

[dosubot]

To address the issue where the BM25 Retriever does not utilize MetadataMode.EMBED or a user-provided option for metadata handling, you can extend the BM25Retriever class to support metadata. This involves modifying the retriever to include an additional parameter for metadata handling and ensuring that both the BM25 and vector index retrievers handle metadata consistently.

Here's a concise approach to modify the BM25Retriever:

1. Extend BM25Retriever: Add a use_metadata boolean flag to indicate whether to use metadata.
2. Uniform Metadata Handling: Ensure the vector index retriever also supports metadata in a similar manner.
3. Metadata Preprocessing: Implement a step to combine text with metadata before indexing and retrieval.
4. Query Expansion: Incorporate metadata into queries for consistent retrieval.
5. Custom Retrieval Logic: Adjust the retrieval methods to consider metadata in scoring.
6. Hybrid Model Integration: Ensure metadata handling is consistent across both retrievers.

Example modification for BM25Retriever:

class BM25Retriever(BaseRetriever):
    def __init__(
        self,
        use_metadata: bool = False,  # New parameter
        # other parameters
    ) -> None:
        self._use_metadata = use_metadata
        # Initialization code
        self._corpus = [self._build_corpus_entry(node) for node in self._nodes]
        # Additional initialization

    def _build_corpus_entry(self, node: BaseNode) -> List[str]:
        content = self._tokenizer(node.get_content())
        if self._use_metadata and node.metadata:
            metadata_content = ' '.join([f"{k}:{v}" for k, v in node.metadata.items()])
            content += self._tokenizer(metadata_content)
        return content

This approach ensures that metadata is consistently handled across both the BM25 and vector index retrievers, improving the accuracy and relevance of search results in a hybrid retrieval model.

Details

Sources

• llama-index-integrations/retrievers/llama-index-retrievers-bm25/llama_index/retrievers/bm25/base.py
• docs/docs/module_guides/indexing/vector_store_index.md
• llama-index-legacy/llama_index/legacy/indices/vector_store/base.py
• llama-index-legacy/llama_index/legacy/extractors/metadata_extractors.py
• llama-index-core/llama_index/core/postprocessor/init.py
• llama-index-integrations/vector_stores/llama-index-vector-stores-vespa/README.md
• llama-index-legacy/llama_index/legacy/vector_stores/utils.py
• llama-index-core/llama_index/core/extractors/metadata_extractors.py
• llama-index-legacy/tests/extractors/test_metadata_extractor.py
• docs/docs/examples/vector_stores/WeaviateIndex_metadata_filter.ipynb
• docs/docs/examples/vector_stores/Qdrant_metadata_filter.ipynb
• llama-index-legacy/llama_index/legacy/embeddings/openai.py
• llama-index-legacy/llama_index/legacy/retrievers/bm25_retriever.py
• llama-index-integrations/retrievers/llama-index-retrievers-mongodb-atlas-bm25-retriever/llama_index/retrievers/mongodb_atlas_bm25_retriever/base.py

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.