Claude Academy
Loading

A multi-search RAG pipeline

Lesson 397 min

When you have both semantic search (vector embeddings) and lexical search (BM25) working independently, the next step is combining them into a unified search pipeline. This hybrid approach leverages the strengths of both methods to deliver more accurate results.

The user question about INC-2023-Q4-011 runs through semantic and lexical search in parallel, then their results merge.

Building a Unified Interface

Both search implementations share nearly identical APIs - they both have add_document() and search() methods. This consistency makes it straightforward to wrap them in a single Retriever class that coordinates between the two approaches.

VectorIndex and BM25Index each have add_document, which adds a text chunk, and "search", which returns chunks and scores.

The Retriever acts as a coordinator that:

  • Receives a user's question
  • Forwards it to both the VectorIndex and BM25Index
  • Collects results from both systems
  • Merges the results using a ranking algorithm

Diagram of the Retriever: a user question goes to both the VectorIndex and the BM25Index, whose results are merged.

Reciprocal Rank Fusion

The challenge lies in merging results from different search methods. Each system returns results with different scoring mechanisms, so you can't simply combine scores directly. Instead, we use a technique called Reciprocal Rank Fusion (RRF).

VectorIndex and BM25Index result tables for the example below, each with a score column, such as 0.234 and 0.184 at rank 1.

Here's how RRF works with a practical example. Suppose your VectorIndex returns results ranked as: Section 2, Section 7, Section 6. Meanwhile, your BM25Index returns: Section 6, Section 2, Section 7.

The ranked results from VectorIndex and BM25Index combined into one table of each text chunk's rank from both indexes.

To merge these results, you create a combined table showing each text chunk's rank from both systems:

Table of each section's ranks from vector and BM25 with its RRF score, noting a higher score means a more relevant chunk.

The RRF formula calculates a score for each document:

RRF_score(d) = Σ(1 / (k + rank_i(d)))

Where k is a constant (typically 60, though 1 works well for clearer results) and rank_i(d) is the rank of document d in the i-th ranking system.

Reciprocal Rank Fusion formula and a table of three sections' vector and BM25 ranks and score, where higher is more relevant.

For each text chunk, you calculate:

  • Section 2: 1.0/(1+1) + 1.0/(1+2) = 0.833
  • Section 7: 1.0/(1+2) + 1.0/(1+3) = 0.583
  • Section 6: 1.0/(1+3) + 1.0/(1+1) = 0.75

After sorting by score, the final ranking becomes: Section 2 (first), Section 6 (second), Section 7 (third).

Table of each text chunk's rank from vector, rank from BM25, score and final rank, where a higher score is more relevant.

Implementation

The Retriever class implementation is straightforward:

python
class Retriever:
    def __init__(self, *indexes):
        self._indexes = list(indexes)
    
    def add_document(self, document):
        for index in self._indexes:
            index.add_document(document)
    
    def search(self, query_text, k=1, k_rrf=60):
        # Get results from all indexes
        all_results = []
        for idx, results in enumerate(all_results):
            for rank, (doc, _) in enumerate(results):
                # Track document ranks across systems
                # Apply RRF formula
                # Return merged, sorted results

Retriever class in a notebook that raises errors for no indexes, non-string query text, non-positive k and negative k_rrf.

The key insight is that the RRF algorithm creates a unified ranking by considering how well each document performs across all search systems, rather than relying on any single scoring method.

Testing the Hybrid Approach

When testing with a query like "what happened with INC-2023-Q4-011?", the hybrid approach delivers significantly better results than either method alone. Instead of getting unexpected results from pure vector search, you now get the most relevant cybersecurity incident report first, followed by related software engineering content.

Notebook output: Section 10 Cybersecurity Analysis scores about 0.0325, Section 2 about 0.0320, Section 5 about 0.0308.

Extensibility

The beauty of this design is its modularity. Since each search index implements the same interface (add_document() and search()), you can easily add new search methodologies to the system. Whether it's a different embedding model, a specialized domain search, or any other retrieval technique, as long as it follows the established API, it integrates seamlessly into the hybrid pipeline.

Retriever sends the question to VectorIndex, BM25Index and a third box labeled ???Index, then merges the two result sets.

This hybrid search approach represents a significant improvement in retrieval accuracy by combining the semantic understanding of vector search with the precise keyword matching of lexical search, all unified through the mathematically sound RRF ranking algorithm.