Evaluation Frameworks: RAGAS & LLM-as-Judge
A prompt tweak shipped two weeks ago — did it help or quietly make things worse? Turning that question from a guess into a measurement.
Did That Prompt Change Actually Help?
Two weeks ago, someone tweaked the refund worker's eligibility-check reasoning to reduce false denials — customers wrongly told they didn't qualify for a refund they were actually entitled to. Nobody can say today whether it worked. A few conversations were spot-checked manually and looked fine, but "looked fine on the handful we happened to read" isn't a measurement, it's a coin flip dressed up as due diligence. Tracing, from the last guide, shows what the system did on any one request. It says nothing about whether a systematic change made things better or worse across the full range of what the system handles.
RAGAS, Generalized Beyond RAG
Phase 6 introduced RAGAS for measuring a RAG pipeline's faithfulness and relevance. The same discipline — RAGAS as a framework, faithfulness and relevance as metrics — generalizes to the whole agent system's outputs, not just retrieval-grounded answers: does the refund agent's final response to a customer accurately reflect what its tool calls actually found (faithfulness, applied to tool outputs instead of retrieved chunks), and does it actually answer what the customer asked (relevance). The mechanism doesn't change; what's being measured against just widens from "retrieved context" to "everything the agent observed during its run."
LLM-as-Judge: Scoring at a Scale Humans Can't Match
Manually reading every conversation doesn't scale past a handful of spot checks. LLM-as-Judge uses a model — often a stronger or differently-calibrated one than the system being evaluated — to score outputs against an explicit rubric, automatically, across as many examples as needed:
Run the same eval set — a curated collection of representative refund scenarios, the same 30-to-50-question discipline from Phase 6's regression suite — against both the old and new eligibility-check prompt, score both with the same judge and rubric, and "did this change help" becomes a comparison of two numbers instead of a handful of manually re-read transcripts.
The Judge Isn't Infallible — Design the Rubric Around That
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., 2023) studied this pattern's actual reliability and found real, systematic biases worth designing around rather than ignoring: verbosity bias (judges tend to rate longer answers more favorably, independent of whether the extra length adds correctness), position bias (when comparing two answers side by side, the judge can favor whichever one appears first), and self-preference bias (a judge model can rate outputs from its own model family more favorably than equally good outputs from a different one). None of these make LLM-as-Judge unusable — they make an unexamined rubric unreliable. Practical mitigations: score on explicit, narrow criteria ("did the response correctly state the refund window," not "is this a good response") rather than open-ended quality, randomize answer order when comparing two outputs, and periodically spot-check the judge's scores against actual human judgment to catch drift.
A rubric that just asks a judge model "rate this response's quality from 1-10" is exactly the setup MT-Bench's verbosity-bias finding warns against — a longer, more hedged answer can score higher than a shorter, more correct one for reasons that have nothing to do with actual quality. Specific, checkable criteria produce far more trustworthy scores than open-ended ones.
Check yourself
A team evaluates two versions of the refund agent's response using an LLM judge with the rubric 'rate the response's overall quality from 1-10.' The newer version, which is longer and more hedged but not more accurate, scores higher. What does MT-Bench's research suggest is the most likely explanation?
What's Next
Measuring quality answers "is it good." It says nothing about "what does it cost to run" — a separate, increasingly urgent question once a beta of 50 customers becomes the full customer base.
Frequently asked questions
Do I need a human-labeled eval set to use LLM-as-Judge, or can the judge create its own ground truth?
A judge still needs something to compare against for many metrics — for faithfulness, that's the retrieved or observed context, not a separate ground-truth label. For metrics like 'was this refund decision correct,' you generally do want a small set of human-verified correct answers, at least initially, both to trust the judge's calibration and to periodically re-check it hasn't drifted.
How often should the regression eval suite run?
At minimum, on every prompt or model change before it ships — the same way a test suite runs before a code merge. Some teams also run it on a schedule against production traffic samples, to catch quality drift that shows up gradually rather than from any single deliberate change.
Is LLM-as-Judge a replacement for human review entirely?
No — it's a scale multiplier, not a replacement. Human review remains valuable for catching failure modes nobody anticipated well enough to write a rubric for, and for periodically validating that the judge's scores still track real human judgment. The realistic pattern is LLM-as-Judge for continuous, high-volume regression testing, with periodic human spot-checks keeping the judge honest.