Keyword Search vs Vector Search: BM25, Embeddings, and Dense Retrieval

The previous article turned documents into searchable chunks. Now the question is: when a user asks something, how does the system decide which chunks look relevant?

1. Retrieval is a ranking problem

Given a query (q), a corpus of chunks (D), and a cutoff (k), a retriever returns an ordered list:

$R_q^k = [d_1,d_2,\ldots,d_k]$

The system does not prove that these chunks are correct evidence. It assigns scores and returns the highest-ranked candidates. A candidate can be topically similar yet have the wrong year, entity, metric, or unit.

2. Keyword retrieval and the inverted index

Keyword search usually starts with an inverted index. Instead of storing only “document → words,” it also stores “term → documents containing the term.”

angola     → chunk 17, chunk 84, chunk 203
production → chunk 12, chunk 84, chunk 611
2017       → chunk 84, chunk 91, chunk 430

At query time, the system can quickly find chunks containing the requested terms. This is especially useful for names, identifiers, numbers, abbreviations, legal phrases, and rare technical vocabulary.

Keyword search is not simply Boolean matching. A ranking function decides which matching chunks should appear first.

3. What BM25 is measuring

BM25 is a widely used lexical ranking function. Its intuition is easier than its formula:

• a term matters more when it appears in the chunk;
• repeated occurrences help, but with diminishing returns;
• rare query terms are usually more informative than common ones;
• unusually long chunks are normalized so they do not win merely by containing more words.

A common form is:

Here (f(t,d)) is the frequency of term (t) in chunk (d); (k_1) controls term-frequency saturation; (b) controls document-length normalization. The exact implementation varies, so BM25 scores should not be treated as probabilities.

4. Dense retrieval and embeddings

Dense retrieval uses an embedding model to map queries and chunks into vectors:

query q → e(q)
chunk d → e(d)

The chunk embeddings are normally computed in advance and stored in a vector index. At query time, the query embedding is compared with them using cosine similarity, dot product, or another distance function.

Dense retrieval can connect paraphrases. A query containing “income after expenses” may retrieve a chunk containing “net profit” even when the exact words do not overlap.

5. Why a bi-encoder scales

Dense retrieval commonly uses a bi-encoder: the query and each chunk are encoded independently. Because chunk vectors are reusable, the system can search a large vector index efficiently.

The trade-off is that the query and chunk do not interact token by token during initial scoring. Their relationship is compressed into two fixed vectors. This is fast, but it can miss fine distinctions such as negation, exact year, units, or which entity owns a value.

6. Their failure modes are complementary

Neither column is universally better. The target questions, corpus, language, and document structure determine which signal is useful.

7. A concrete annual-report example

Question: “In 2017, what share of equity liquid production outside Norway came from three blocks in Angola?”

• BM25 may reward exact matches for 2017, Angola, and equity liquid production.
• Dense retrieval may find a table whose header uses a paraphrase not present in the query.
• Dense retrieval may also rank an almost identical 2016 table highly.
• BM25 may miss the correct row if OCR changed a name or the query uses a synonym.

This is why semantic similarity is not evidence correctness. Retrieval creates candidates; later stages still need metadata checks, fusion, reranking, and evaluation.

8. What should be indexed?

For lexical search, preserve meaningful tokens, numbers, table headers, and domain terms. For dense search, construct text that gives the embedding model enough context: section title, table title, row labels, units, and selected metadata.

Do not silently append every metadata field. Test whether it helps retrieval or merely makes unrelated chunks look alike.

9. How should the two retrievers be compared?

Use the same queries, gold evidence, chunk collection, and cutoff values. Compare Recall@k and ranking metrics, then inspect disagreements:

1. evidence found only by BM25;
2. evidence found only by dense retrieval;
3. wrong-year or wrong-entity near misses;
4. failures caused by extraction rather than retrieval.

The disagreement set explains why a method works and motivates the next design step: combine complementary candidate lists, then rerank them more carefully.

Further reading

• Elasticsearch: BM25 similarity
• Sentence Transformers: semantic search
• Sentence Transformers: retrieve and rerank