Reviewed 25 Sept 2026 / 7 min / Retrieval & RAG 5 of 6

Hybrid search: the basics

Semantic search fails predictably on the exact tokens people actually search for — product codes, names, error strings — because an embedding model has no special respect for a rare word. Running a keyword index alongside the vector one and merging the two rankings fixes that, and reciprocal rank fusion merges them without needing the two score scales to mean anything to each other.

Hybrid search: the basics

A five-minute read.

The RAG post ended by naming two refinements and declining to explain either. This is the first of them, and it also pays off a line from the vector databases post: keyword search still wins for product codes, names, and rare terms. Here’s why that’s true, and what to do about it.

Where semantic search actually loses

An embedding model maps text to a region of space by meaning. That’s exactly what you want for “how do I reset my password” matching “password reset instructions”. It is exactly what you don’t want when the query is the rare token.

Ask a vector index for ERR_5521. The model has almost certainly never seen that string in training, so it tokenizes into fragments and produces a vector that means roughly “some kind of error code.” That vector is genuinely close to every chunk discussing errors, logging, and retries — and only incidentally close to the one chunk that actually names ERR_5521.

flowchart LR
    Q["query: ERR_5521"] --> M["embedding model"]
    M --> V["#91;0.31, -0.08, ...#93;<br/>a generic 'error-ish' vector"]
    V --> R["top hits: error handling,<br/>logging, retry policy"]
    R -.the chunk you wanted.-> X["the one chunk naming<br/>ERR_5521 — rank 40"]

I saw this land badly on an internal docs search. Semantic-only, and it demoed well — every vague question about onboarding or expense policy came back with the right page. Then a support engineer pasted an error code into it on the first day, got three pages of general troubleshooting advice, and went back to grepping the wiki. The page documenting that exact code did exist. It was sitting somewhere around rank 40, below every chunk that merely mentioned errors in passing.

The same failure hits person names, version numbers, SKUs, function names, and legal citations. These are precisely the queries where the user knows exactly what they want and expects an exact match, which makes the failure feel much worse than a fuzzy miss on a fuzzy question.

What BM25 counts

The fix is old technology. BM25 is a keyword ranking function that scores a chunk against a query using three ingredients, none of which involve a model:

• Term frequency. How often the query’s terms appear in this chunk, with diminishing returns, so a chunk saying ERR_5521 twelve times doesn’t beat a relevant one saying it twice by a factor of six.
• Inverse document frequency. How rare each term is across the whole corpus. Terms appearing in almost every chunk contribute almost nothing; a term appearing in one chunk contributes enormously.
• Length normalization. A long chunk has more chances to contain your term by accident, so its matches are discounted.

That middle ingredient is the whole story. Rarity is what a vector index throws away and what BM25 weighs most heavily — the two methods fail and succeed on opposite inputs. (BM25 has tuning knobs of its own, k1 and b, controlling saturation and length normalization. The defaults are fine for a long time.)

Running both

Hybrid search means keeping two indexes over the same chunks, querying both, and merging the rankings.

flowchart LR
    Q["user question"] --> V["vector index<br/>ranks by meaning"]
    Q --> K["keyword index<br/>ranks by rare exact terms"]
    V --> A["ranked list A"]
    K --> B["ranked list B"]
    A --> F["fusion"]
    B --> F
    F --> R["one merged ranking"]

The chunks themselves don’t change — this is still the output of the chunking post, indexed twice. Everything downstream, including the prompt assembly from the RAG post, sees one list and doesn’t need to know it came from two.

The fusion problem, and why ranks beat scores

The obvious merge is to add the two scores. It doesn’t work, because the scores aren’t in the same units and never will be.

Cosine similarity is bounded — in practice a number between 0 and 1, roughly comparable from one query to the next. BM25 is unbounded and depends on corpus statistics and query length: a score of 14 might be an excellent match for a three-word query and a mediocre one for a ten-word query. Min-max normalizing each list per query is the usual patch, and it’s fragile, because the range you’re normalizing against depends on whatever happened to be retrieved for that particular query.

Reciprocal rank fusion sidesteps this entirely by throwing the scores away and keeping only the positions.

def reciprocal_rank_fusion(
    rankings: list[list[str]],
    k: int = 60,
) -> dict[str, float]:
    scores: dict[str, float] = {}
    for ranking in rankings:
        for rank, chunk_id in enumerate(ranking, start=1):
            scores[chunk_id] = scores.get(chunk_id, 0.0) + 1 / (k + rank)
    return scores

A chunk earns 1 / (k + rank) from each list it appears in, and the contributions add up. With k = 60, first place is worth 1/61 and second is worth 1/62 — nearly identical — so no single list can dominate on the strength of one confident top hit. What wins is appearing respectably in both.

Chunk B is first overall despite topping neither list. That’s the intended behaviour: agreement between two methods that fail differently is stronger evidence than a single first-place finish.

When to weight instead

RRF gives both retrievers an equal vote. Sometimes you know that’s wrong — a codebase search should lean lexical, a support-article search should lean semantic. The alternative is weighted score fusion: normalize each list, then combine with alpha * vector + (1 - alpha) * keyword.

This buys you a dial, at the cost of reintroducing the normalization fragility above and adding a parameter you now have to justify. Tuning alpha by intuition is guessing; tuning it properly needs a labeled set of queries and a way to score the results, which is its own topic and the next post in this series.

Start with RRF. Move to weights when you can measure that the move helped.

Two things that bite beginners

Two indexes over one corpus means two things to keep in sync. Re-chunking rebuilds both, same as the chunking post’s warning about re-embedding. Delete a document from one and not the other and it keeps surfacing as a phantom result from the half you forgot.

Hybrid can score worse than the better half alone. Fusing a strong retriever with a weak one lets the weak list drag irrelevant chunks into the merged top-k, and RRF’s equal vote makes that easy to do by accident. I’ve shipped exactly that: BM25 bolted onto a corpus of narrative prose with almost no rare tokens in it, where every keyword list was close to noise and got an equal vote anyway. The merged results were worse than vector alone, and I only found that out because I went back and measured. Hybrid search is a hypothesis about your corpus, not a free upgrade.