Hybrid Search & Re-ranking
Why pure semantic search misses exact error codes and product IDs, and how combining it with keyword search plus a cross-encoder re-ranker fixes it.
Fast, and Still Sometimes Wrong
Search latency is fixed — HNSW brought a million-chunk index down from 3 seconds to under 150ms. A new complaint shows up anyway: a user searches for "error code E402" and gets back chunks about general troubleshooting instead of the one paragraph that actually documents E402. The embedding model captured the topic (troubleshooting) but not the specific token (E402) — semantic search is genuinely good at conceptual similarity and genuinely weak at exact, rare-term matching, because an embedding blurs specific tokens into a broader meaning representation. A keyword search over the same corpus would have found the exact string "E402" instantly. Neither approach alone is sufficient; the fix is using both.
BM25: What Semantic Search Is Bad At
BM25 is the modern standard keyword-ranking algorithm — a refinement of decades-old term-frequency scoring that ranks documents by how often and how distinctively a query's exact terms appear in them, weighted so rare terms (like an error code) count for more than common ones. It has no concept of meaning at all: it wouldn't connect "duplicate transaction" with "charged twice" the way an embedding would. That's precisely the complementary weakness — BM25 is strong exactly where semantic search is weak, and vice versa.
Hybrid Search: Running Both and Merging
Hybrid search runs a BM25 query and a semantic (vector) query against the same request, then merges the two ranked lists into one — typically using Reciprocal Rank Fusion (RRF), which combines results based on each document's rank position in each list rather than trying to reconcile two incompatible scoring scales (BM25 scores and cosine-similarity scores aren't directly comparable, so merging by rank sidesteps that problem entirely).
For the E402 query, the fix is direct: the keyword search surfaces the exact-match document that semantic search alone missed, while the semantic search still catches queries phrased conceptually with no exact term overlap at all — a user who writes "getting an odd validation failure" instead of quoting the error code by name.
Pinecone's guide to hybrid search covers the mechanics of combining sparse (BM25-style) and dense (embedding-based) retrieval in more implementation depth, including how to weight one against the other.
Re-ranking: A Second, More Expensive Pass
Hybrid search improves what gets retrieved. Re-ranking improves the order of what's already been retrieved, using a more expensive model that wouldn't be practical to run over the whole index in the first place.
The embedding models used for retrieval are bi-encoders: they encode the query and each document independently into vectors, which is what makes pre-computing and indexing millions of document embeddings possible — the query only needs to be embedded once per search, then compared against vectors that already exist. A cross-encoder, by contrast, takes the query and a single candidate document together as one input and directly scores how well they match — a fundamentally more accurate comparison because the model can attend across both texts jointly, at the direct cost of having to run this expensive joint computation separately for every candidate, which makes it far too slow to run over a million-chunk index directly. Sentence-BERT (Reimers & Gurevych, 2019) is the paper that formalized this bi-encoder/cross-encoder distinction and remains the standard reference for why production retrieval systems use both in sequence rather than picking one.
The standard pattern is therefore two stages: retrieve a broad set of candidates cheaply (hybrid search, top 50 or so), then re-rank only that small candidate set with a cross-encoder for a much higher-precision final ordering before the top few go into the model's context.
Check yourself
Why isn't a cross-encoder used directly for the first-pass retrieval over a million-document index, instead of retrieving with embeddings first and re-ranking a smaller set afterward?
What's Next
Hybrid search and re-ranking assume the user's query itself is a reasonable starting point for retrieval — but sometimes the biggest gap is between how a user phrases a question and how the answer is actually worded in the source documents. That's a query problem, not a ranking problem, and it's the last piece of this phase.
Frequently asked questions
Is hybrid search always better than semantic search alone?
For most real-world corpora, yes, or at worst neutral — the cost is running two searches instead of one, which is a modest latency and infrastructure overhead, not a quality tradeoff. It matters most when a corpus contains exact identifiers, codes, or rare technical terms; a corpus of purely conversational, free-form text sees a smaller benefit but rarely a downside.
How many candidates should go into the re-ranking stage?
Enough to reasonably contain the true best answer, small enough to stay fast — 20 to 100 candidates is a common range in practice. Too few risks the re-ranker never seeing the right document at all; too many erodes the latency benefit of doing cheap retrieval first.
Can re-ranking replace good chunking and embedding strategy from the previous phase?
No — re-ranking can only reorder what retrieval actually surfaced. If the right chunk was never in the candidate set in the first place (a chunking or embedding problem), no amount of re-ranking recovers it. Re-ranking improves precision among retrieved candidates; it doesn't fix a recall problem upstream of it.