Why Hybrid Retrieval and Reranking? From Candidates to Better Evidence

The previous article showed that BM25 and dense retrieval fail differently. A practical system can use both instead of forcing one method to solve every query.

1. Why retrieval is often split into stages

A production corpus may contain millions of chunks. A highly expressive model cannot compare the query with every chunk at full cost.

The usual compromise is a funnel:

large corpus
   ↓ fast candidate retrieval
Top-50 or Top-100
   ↓ stronger reranker
Top-5 or Top-10
   ↓ context construction
generator

The first stage optimizes coverage and speed. The second stage optimizes ordering precision inside the candidate set.

2. What “hybrid retrieval” means

Hybrid retrieval combines results from two or more retrieval methods. A common design runs BM25 and dense retrieval in parallel:

• BM25 contributes exact terms, numbers, identifiers, and rare names;
• dense retrieval contributes paraphrases and semantic similarity;
• metadata filters enforce hard constraints such as year or document type.

Hybrid retrieval is not a new truth detector. It creates a larger or better candidate pool from complementary signals.

3. Why scores cannot simply be added

A BM25 score and a cosine-similarity score live on different scales. Their distributions can also change by query. Adding raw scores may let one retriever dominate for accidental numerical reasons.

There are three common approaches:

1. normalize scores before combining them;
2. learn a fusion model from relevance labels;
3. combine ranks instead of raw scores.

The third approach leads to Reciprocal Rank Fusion.

4. Reciprocal Rank Fusion (RRF)

RRF assigns each document a score based on its position in every ranked list:

(R) is the set of retrievers, and (k) is a constant that reduces the effect of extreme top ranks. A result appearing near the top of several lists receives a strong combined score.

RRF is attractive because it does not require comparable raw scores. But it ignores how far apart raw relevance scores were and needs enough depth from each retriever to include useful evidence.

5. Deduplication and provenance come before reranking

After fusion, the same source may appear several times as overlapping chunks. If duplicates occupy the entire Top-k, the generator sees less independent evidence.

A fusion layer should therefore preserve source IDs, merge exact duplicates, control near-duplicates, retain the best provenance, and avoid removing distinct rows that happen to share text.

6. What a reranker changes

A common neural reranker is a cross-encoder. Unlike a bi-encoder, it reads the query and candidate together:

[query ; candidate] → Transformer → relevance score

Joint attention lets it examine exact relationships between the two texts: which year modifies which metric, whether a statement is negated, and whether the candidate actually answers the query.

Cross-encoders are more expensive because every query–candidate pair requires a model pass. This is why they rerank tens or hundreds of candidates rather than searching the complete corpus.

7. Candidate recall is the reranker’s ceiling

The most important limitation is simple:

A reranker can move a retrieved item upward, but it cannot recover evidence that never entered the candidate set.

If the correct table is absent from Top-100, perfect reranking still fails. Measure first-stage Recall@k before blaming the reranker.

8. Reranking is not evidence verification

A relevance score answers something like “How well does this candidate match the query?” It does not automatically prove that the number is correct, that every required evidence item is present, or that a citation supports the final claim.

Use metadata constraints and later verification for conditions that must be exact. For the annual-report example, a wrong-year table may remain semantically relevant; year consistency should be checked explicitly.

9. A practical design

query
 ├─ BM25 Top-50
 ├─ dense Top-50
 └─ metadata constraints
          ↓
      RRF + deduplication
          ↓
     60 unique candidates
          ↓
    cross-encoder reranker
          ↓
  evidence-aware Top-5 context

Tune candidate depth, fusion, reranker cutoff, and context size separately. Increasing every number usually increases latency and noise without guaranteeing better evidence.

10. How to evaluate the pipeline

Evaluate each boundary rather than only the final answer:

An ablation comparing BM25, dense, hybrid, and hybrid-plus-reranker reveals whether added complexity earns its cost.

Further reading

• Cormack, Clarke & Büttcher (2009): Reciprocal Rank Fusion
• Sentence Transformers: Retrieve & Re-Rank
• Sentence Transformers: Cross-Encoders