RAG Evaluation, Debugging & Architecture Selection
How to measure whether a RAG pipeline is actually working, find which stage is failing when it isn't, and pick the right architecture instead of guessing.
Four Architectures, One Open Question
The refund bot's retrieval now spans four distinct architectures: hybrid vector search for direct lookups, GraphRAG for multi-hop relationship questions, agentic retrieval for open-ended multi-step ones, and KAG where auditability matters more than flexibility. What's missing is a way to actually know whether any of them is working — and when one underperforms, whether the problem is retrieval finding the wrong content or generation misusing content that was actually correct. Without measurement, "which architecture should handle this kind of question" is a guess dressed up as an engineering decision.
What to Actually Measure
RAG evaluation splits cleanly into retrieval quality and generation quality, and conflating the two is the single most common reason teams can't debug a bad answer:
- Context precision / recall — of the chunks retrieved, how many were actually relevant (precision), and of the relevant chunks that existed, how many got retrieved (recall)? This measures retrieval alone, independent of what the model does with what it's given.
- Faithfulness — does the generated answer's claims actually match what's present in the retrieved context, or does it add, distort, or invent something not supported by it? This is the direct, measurable version of the citation-verification idea from Phase 4 — faithfulness is what a guardrail checking citations is actually trying to enforce.
- Answer relevance — does the response actually address the question asked, independent of whether it's faithful to the retrieved context? A perfectly faithful answer to a question the user didn't ask is still a failure.
RAGAS is the standard open-source framework for computing these metrics automatically, typically using an LLM as the judge scoring each dimension — the same LLM-as-judge pattern Phase 9 covers in full for evaluation more broadly. Running these metrics isn't a one-time check; the useful pattern is a regression suite of representative questions per architecture, scored on every pipeline change, the same way you'd run a test suite against application code.
Debugging: Which Stage Actually Failed
A wrong answer has exactly two possible root causes, and they need opposite fixes — which is why separating retrieval metrics from generation metrics above isn't optional:
- Retrieval failed. The right content either doesn't exist in the corpus at all, or exists but wasn't surfaced — low context recall is the signal. The fix lives in the phases before this one: chunking strategy, embedding model choice, hybrid search, or reaching for GraphRAG if the question was actually multi-hop and the architecture never had a chance.
- Generation failed despite good retrieval. The right chunks were retrieved (high context precision/recall), but the answer is unfaithful to them anyway — misreading a chunk, ignoring it in favor of trained-in knowledge, or answering a slightly different question than what was asked. The fix lives in prompting discipline (Phase 2's grounding instructions) or model choice, not in retrieval at all.
Seven Failure Points When Engineering a RAG System catalogs concrete, real-world versions of both categories — content genuinely missing from the source corpus, retrieval technically finding a chunk but failing to rank it highly enough to make the cut, and the model receiving the right context but failing to extract or use it correctly. Reading it alongside your own RAGAS scores is a useful way to build intuition for which failure category a specific bad answer actually falls into, rather than guessing.
Choosing an Architecture Instead of Guessing
| Question shape | Architecture | Why |
|---|---|---|
| Direct fact lookup ("what's the refund window?") | Hybrid vector RAG | Fastest, cheapest, sufficient when the answer lives in one passage |
| Relationship across known entities ("who qualifies given plan X and bonus Y?") | GraphRAG | The connection itself is the thing being asked for |
| Open-ended, unpredictable multi-step ("compare old and new policy") | Agentic RAG | The retrieval sequence can't be fixed in advance |
| High-stakes, must be auditable (compliance, legal, financial) | KAG | Predictable, structured reasoning matters more than flexibility |
This table is a starting point, not a substitute for the measurement above — the honest process is routing a representative sample of real user questions to each candidate architecture, scoring them with the metrics from this guide, and letting the numbers (not the table) settle close calls.
Check yourself
A RAG pipeline gives a wrong answer. Context recall is high (the right chunks were retrieved) but faithfulness is low. What does this combination indicate, and where's the fix?
Closing This Phase
Naive RAG's accuracy ceiling from the start of this phase is gone, replaced with a real decision framework: hybrid search for direct lookups, GraphRAG for relationships, agentic retrieval for unpredictable multi-step questions, KAG where auditability is non-negotiable, and RAGAS-based evaluation in place to measure which one is actually working rather than assuming. The refund bot now correctly answers the Premium-plan-and-loyalty-bonus question through GraphRAG, handles the old-versus-new policy comparison through agentic retrieval, and the team has a regression suite that catches quality drops before they ship, not after a customer notices.
Agentic RAG's retrieval-as-a-tool pattern from the second guide in this phase is also a preview of where this roadmap goes next: an agent that can call retrieval is one step away from an agent that can call anything — calculations, other APIs, other agents. Phase 7 picks that up directly.
Frequently asked questions
Is RAGAS the only option for RAG evaluation?
It's the most widely adopted open-source framework specifically built around the faithfulness/relevance/precision/recall metrics covered here, but it isn't the only tool — some teams build custom LLM-as-judge rubrics for domain-specific quality dimensions RAGAS doesn't cover out of the box. RAGAS is a strong default starting point rather than the only correct choice.
How many test questions are enough for a meaningful RAG regression suite?
Fewer than you'd think, done well — 30-50 carefully chosen questions covering each architecture and each known failure mode usually surfaces regressions faster than a much larger but less deliberately curated set. Coverage of question *types* (direct lookup, multi-hop, comparison) matters more than raw volume.
Should every RAG system eventually need all four architectures from this phase?
No — most systems are well served by hybrid vector RAG alone, and adding GraphRAG, agentic retrieval, or KAG is justified specifically by measured failure on the question types those architectures target, not by default. Reaching for all four upfront adds real engineering and operational cost for problems that may never actually show up in your corpus or user questions.