GraphRAG: Retrieval Over Relationships, Not Just Chunks
Why a fast, well-tuned vector index still fails on questions that span multiple documents, and how building a knowledge graph fixes it.
A Question With No Single Answer Chunk
The refund bot's retrieval is fast, hybrid, and re-ranked. It still fails, reliably, on one specific kind of question: "Which customers who upgraded to Premium in the last 90 days also qualify for the loyalty refund bonus, and what policy actually applies to them?" Retrieval dutifully returns a chunk about the Premium plan's upgrade terms and a separate chunk about loyalty bonus eligibility — both genuinely relevant, both correctly ranked — but no single chunk states the combined policy, because no one ever wrote a document that connects those two facts. Every technique from the last two phases assumed the answer lived somewhere retrievable as a passage of text. This question's answer doesn't live in any one passage — it lives in the relationship between two separate facts.
This is a multi-hop question, and it's a structural ceiling for flat vector retrieval, not a tuning problem. No amount of better chunking, better embeddings, or better re-ranking fixes it, because the fix those techniques offer is "retrieve the right existing passage" — and the right passage doesn't exist.
Retrieval Over a Graph Instead of a Flat Index
GraphRAG replaces (or supplements) the flat chunk index with a knowledge graph: entities (Premium plan, loyalty bonus, refund policy, customer segment) as nodes, and the relationships between them (Premium plan grants loyalty bonus eligibility, loyalty bonus modifies refund policy) as edges, extracted from the same source documents using an LLM during ingestion. Answering a multi-hop question then means traversing the graph — following the Premium-plan node to its eligibility edge, then following that to the bonus node, then to the modified-policy node — instead of hoping a single retrieved chunk already contains the connected fact.
Local Search vs. Global Search
From Local to Global: A GraphRAG Approach to Query-Focused Summarization (Microsoft Research) — the paper that popularized this pattern at scale — draws a useful distinction between two different retrieval modes over the same graph:
- Local search answers questions about specific entities and their immediate relationships (the loyalty-bonus question above is this kind: traverse a few specific nodes and edges to a precise answer).
- Global search answers broader, corpus-wide questions ("what are the main categories of customer complaints this quarter?") by first summarizing clusters — "communities" — of related entities in the graph, then reasoning over those summaries rather than the raw graph. This is the piece flat vector retrieval has essentially no equivalent for: there's no single chunk that summarizes a theme spanning hundreds of documents, but a pre-computed community summary in the graph can.
Microsoft's GraphRAG project is a working open-source implementation of both modes, including the graph-construction pipeline (entity and relationship extraction, community detection, and summarization) — useful to read even if you end up building a lighter-weight version for a narrower use case.
The Cost Side of the Tradeoff
None of this is free. Building the graph means running entity and relationship extraction over the entire corpus during ingestion — a meaningfully more expensive process than chunking and embedding, and one that has to be re-run (at least incrementally) as documents change, layering on top of the freshness discipline from Phase 4. GraphRAG is a specialized tool for a specific failure mode, not a wholesale replacement for the flat retrieval covered in the previous two phases — most single-fact lookup questions are still answered faster and more cheaply by hybrid vector search, and reaching for a knowledge graph to answer "what's your refund window" would be solving a problem that was never there.
Check yourself
Why does better chunking, embedding, or re-ranking fail to fix the Premium-plan-and-loyalty-bonus question, even though each individual fact is retrievable on its own?
What's Next
GraphRAG fixes questions that need connecting specific facts through explicit relationships. Not every hard question fits that shape — some need the model to figure out, step by step, what to look up next based on what it's already found, which is a different kind of flexibility than traversing a pre-built graph.
Frequently asked questions
Does GraphRAG replace the vector index from previous phases?
Not usually — most production systems run both, routing single-fact lookups to fast hybrid vector search and multi-hop or thematic questions to the graph. Building and maintaining a graph for queries flat retrieval already handles well just adds ingestion cost with no accuracy benefit.
How is the knowledge graph actually built from unstructured documents?
Typically an LLM processes each document (or chunk) during ingestion and extracts entities and relationships as structured output — the same structured-output pattern from Phase 2, applied to entity extraction instead of classification. This is why graph construction is expensive: it's an LLM call per document, not a lightweight embedding pass.
Is GraphRAG only useful for questions explicitly about relationships?
Its clearest wins are relationship and multi-hop questions, but the community-summarization side (global search) also helps with broad, thematic questions over a large corpus that no single document addresses — a different failure mode than multi-hop, but one flat retrieval handles just as poorly.