Query Rewriting: Expansion, HyDE & Multi-Query
What to do when the user's own words are the problem — rewriting a query before it ever reaches the index, instead of tuning retrieval further.
When the Query Itself Is the Problem
The index is fast (HNSW), and it handles both exact terms and conceptual matches well (hybrid search plus re-ranking). One more failure mode remains, and it isn't a retrieval-tuning problem at all: a user writes "card charged twice," and the relevant policy document is worded entirely differently — "duplicate transaction dispute resolution." The vocabulary gap is real enough that even a good embedding model places the short, informal query meaningfully far from the formally worded document, and no amount of re-ranking helps if the right document never made it into the candidate set to begin with. The fix has to happen before the query hits the index at all.
Query Expansion: Add What Might Be Missing
The simplest fix adds related terms to the query before searching — synonyms, common rephrasings, or terms an LLM predicts might appear in a relevant document. "card charged twice" expands to something like "card charged twice OR duplicate charge OR duplicate transaction OR billed twice". This widens the net for both keyword and semantic matching, at the cost of some added noise if the expansion terms drift from the user's actual intent.
HyDE: Search With the Answer, Not the Question
Hypothetical Document Embeddings (HyDE), introduced in Precise Zero-Shot Dense Retrieval without Relevance Labels (Gao et al., 2022), takes a genuinely different approach: instead of embedding the user's short question directly, ask an LLM to first generate a hypothetical answer to that question — a plausible-sounding passage that might address it, with no requirement that it be factually correct — and embed that instead.
The insight is subtle but effective: a short question and a long, formally worded answer are structurally dissimilar pieces of text even when they're about the same topic, so their embeddings end up farther apart than you'd want. A hypothetical answer, even a fabricated and possibly wrong one, tends to be worded much more like a real answer document would be — same register, same level of detail, similar vocabulary — so its embedding lands much closer to the real, correct document in vector space. The hypothetical content itself is never shown to the user or trusted as a claim; it exists purely to generate a better-positioned search vector, which sidesteps the exact hallucination concerns covered in Phase 1 since nothing hypothetical ever reaches the final answer.
Multi-Query: Ask It Several Different Ways
Multi-query fan-out generates several reworded variants of the user's question, runs retrieval for each in parallel, and merges the deduplicated results — the same reciprocal-rank-fusion idea from hybrid search, applied across query rewordings instead of across search algorithms:
Original: "card charged twice"
Generated variants:
1. "duplicate charge on my account"
2. "billed multiple times for one order"
3. "unauthorized repeat transaction"
Each variant is likely to overlap in vector space with a slightly different subset of relevant documents, so the union of all three retrievals covers more ground than any single phrasing would alone — at the direct cost of running retrieval multiple times per user query, which is a latency and infrastructure tradeoff worth measuring against the recall improvement it actually buys for a given corpus.
See LangChain's MultiQueryRetriever documentation for a working implementation of this pattern, including how the variant-generation prompt itself is typically structured.
Check yourself
Why does embedding a hypothetical (possibly fabricated) answer, as HyDE does, often retrieve better results than embedding the user's original short question directly?
Fast, Accurate, and Recall-Robust
This phase closes the story that started with a three-second search: the index is fast (HNSW), it catches both exact terms and conceptual matches (hybrid search and re-ranking), and it's robust to a user's specific wording not matching a document's specific wording (query rewriting and HyDE). None of these techniques individually would have been enough — a fast index searching the wrong candidates is still a wrong answer, and a slow index with perfect recall is still a bad user experience.
There's a ceiling to how far tuning a single retrieval pipeline can go, though. Some questions genuinely need multi-hop reasoning across several documents, or a fundamentally different retrieval backend than flat chunks in a vector index — that's where the next phase picks up, with GraphRAG and agentic retrieval architectures.
Frequently asked questions
Should I use query expansion, HyDE, and multi-query all at once?
Rarely all three simultaneously — each adds latency and complexity, and they overlap in what they fix. HyDE and multi-query both specifically target vocabulary-mismatch problems; picking one based on measured retrieval failures on your own corpus is usually more effective than stacking all of them by default.
Does query rewriting add noticeable latency to a request?
Yes — HyDE and multi-query both require an extra LLM call (to generate the hypothetical answer or the query variants) before retrieval even starts, on top of the retrieval itself. This is a real cost worth measuring against the recall improvement for your specific corpus, not something to add reflexively.
Is HyDE useful for queries that already closely match document wording?
Less so — its value is specifically in bridging a vocabulary gap between how users ask and how documents are written. For a corpus where users already tend to phrase things close to the source documents (technical users searching technical documentation, for example), the added latency of generating a hypothetical answer may not pay for itself.