Agentic RAG & KAG
When the fix isn't a better index but letting the model decide what to retrieve next — and the more structured alternative for high-precision domains.
A Question That Needs a Plan, Not Just a Graph
GraphRAG closes the multi-hop gap for questions that fit a relationship the graph already models. A different question exposes a separate limitation: "How does my refund eligibility differ between the old policy and the new one that took effect last month?" Answering this well requires the model to first retrieve the old policy, then retrieve the new policy, then explicitly compare them — a sequence of retrieval steps decided during answering, based on what's been found so far, not a fixed traversal of a pre-built graph. Every RAG architecture so far — flat, hybrid, graph-based — retrieves once (or traverses a fixed structure once) and then generates. This question needs retrieval to happen adaptively, more than once, with the model deciding each step.
Agentic RAG: Retrieval as a Tool the Model Decides to Use
Agentic RAG reframes retrieval from a fixed pipeline stage into a tool the model can call — using exactly the function-calling mechanism from Phase 2 — as many times as it decides it needs, in whatever order the question demands. Instead of "always retrieve top-k, then answer," the loop looks like:
This is the ReAct reasoning loop — covered in full in the next phase — applied specifically to retrieval as the available tool. The model isn't guessing at a single best query upfront; it retrieves, reads what came back, and decides whether it has enough to answer or needs to retrieve again with a refined or entirely different query. For the policy-comparison question, that adaptivity is the entire fix: no single fixed query ("old vs new refund policy") reliably retrieves both documents as well as two deliberate, sequential, targeted retrievals do.
LlamaIndex's guide to building an agent walks through wiring retrieval up as a callable tool in exactly this pattern — a practical starting point before the fuller agent architectures covered in Phase 7.
KAG: The More Structured Alternative
Agentic RAG's flexibility is also its risk: an agent that decides its own retrieval steps can decide badly, and in domains where a wrong or incomplete answer has real consequences — legal, medical, financial, exactly the refund-and-compliance territory this roadmap's scenarios keep landing in — that unpredictability is a real cost, not just an engineering inconvenience. Knowledge-Augmented Generation (KAG), introduced in KAG: Boosting LLMs in Professional Domains via Knowledge Augmented Generation, takes a more constrained approach: combine a knowledge graph's structure (Phase 6's first guide) with explicit logical reasoning steps the system enforces, rather than leaving the retrieval-and-reasoning sequence entirely up to the model's own judgment call at each step.
Where agentic RAG asks "what would the model like to look up next," KAG asks "what does the domain's logical structure require to be looked up, in what order, to answer this class of question correctly" — trading some of agentic RAG's open-ended flexibility for a more auditable, more predictable reasoning path, which matters more in domains where "the model improvised a reasonable-sounding retrieval sequence" isn't good enough.
| Approach | Retrieval decided by | Best for | Cost |
|---|---|---|---|
| Flat/hybrid RAG (Phases 4-5) | Fixed, single retrieval per query | Direct fact lookup | Lowest |
| GraphRAG | Pre-built graph traversal | Multi-hop questions over known relationships | Graph construction cost |
| Agentic RAG | The model, adaptively, per question | Open-ended, unpredictable multi-step questions | Variable — depends how many retrieval steps the model takes |
| KAG | Domain-defined logical structure | High-precision domains needing auditable reasoning | Structure-definition cost upfront |
Check yourself
A financial-compliance question needs a retrieval sequence that must be predictable and auditable, not just eventually correct. Which is the more appropriate architecture, and why?
What's Next
Four RAG architectures now exist side by side — flat/hybrid, graph-based, agentic, and knowledge-augmented — and picking the right one per question, or diagnosing why any of them is underperforming, needs an actual measurement framework rather than intuition. That's the last guide in this phase.
Frequently asked questions
Is agentic RAG always slower than a single-shot retrieval?
Usually, yes — each additional retrieval step is an extra round trip and often an extra LLM call to decide what to retrieve next. That latency cost is exactly why agentic RAG is worth reserving for questions that genuinely need adaptive, multi-step retrieval, rather than applying it as a default to every query flat retrieval already handles in one pass.
Can agentic RAG and GraphRAG be combined?
Yes, and it's a common production pattern — an agent decides when to query the graph (for relationship questions), when to query the flat vector index (for direct lookups), and when to do both and synthesize. The graph becomes one of several tools available to the agent, rather than a separate, standalone architecture.
Does KAG require building a full knowledge graph like GraphRAG does?
Often, yes — KAG builds on graph-structured knowledge as its foundation, then adds explicit logical constraints on top of it. The two aren't mutually exclusive; KAG can be thought of as GraphRAG's graph plus a more disciplined, auditable reasoning layer on top, rather than a wholly separate retrieval backend.