Agent Evaluation: Catching Regressions Before Customers Do
Stop eyeballing responses. Use Spring AI's Evaluator API — RelevancyEvaluator, FactCheckingEvaluator, and a hand-rolled LLM-as-a-judge — to turn answer quality into a CI gate.
The Prompt Tweak Nobody Reviewed
Three phases in, the support agent does a lot: grounded RAG answers, live order lookups, escalation, tools shared over MCP. Then someone on the team tweaks the system prompt to fix an unrelated tone complaint, ships it, and two weeks later a support lead notices the bot has been confidently answering refund-eligibility questions from documents that don't actually cover the customer's tier — a regression nobody caught, because nobody was checking anything except "does it look plausible in a quick manual test."
Every other guide in this roadmap has a test story: JUnit for your Java code, integration tests for your endpoints. RAG and agent responses need the same discipline, they just can't be asserted with assertEquals — there's no single correct string. Spring AI's Evaluator API exists to make "is this answer actually good" a repeatable, automatable check instead of a vibe.
No new build.gradle dependency for RelevancyEvaluator or FactCheckingEvaluator either — both are bundled in the same chat model starter you've had since the first guide, and only need a ChatClient.Builder. The one addition, if your project doesn't already have it, is JUnit's parameterized-test support for the golden-set example in section 5:
// build.gradle — only needed for @ParameterizedTest / @MethodSource
testImplementation 'org.junit.jupiter:junit-jupiter-params'Spring Initializr adds spring-boot-starter-test (which pulls in JUnit 5, including junit-jupiter-params) to every generated project by default, whether or not you explicitly selected it back in the first guide — so this is usually already on your test classpath with nothing to add.
1. The Evaluator Interface
Every evaluator in Spring AI implements the same one-method shape:
@FunctionalInterface
public interface Evaluator {
EvaluationResponse evaluate(EvaluationRequest evaluationRequest);
}EvaluationRequest bundles what you're checking: the original question, the context the answer was supposed to be grounded in, and the response itself. EvaluationResponse comes back with a pass/fail:
EvaluationRequest request = new EvaluationRequest(
userQuestion, // what was asked
retrievedContext, // List<Content> — what the model was given to answer from
generatedResponse // what the model actually said
);
EvaluationResponse response = evaluator.evaluate(request);
boolean passed = response.isPass();Both evaluators Spring AI ships are themselves LLM-as-a-judge: evaluate() makes its own ChatClient call, asking a model to grade the response against a rubric baked into the evaluator's prompt template. You're not writing string-matching assertions — you're delegating "is this good" to a model, the same way a human reviewer would judge it, just automatable and consistent.
2. RelevancyEvaluator: Does the Answer Address the Question?
RelevancyEvaluator relevancyEvaluator = new RelevancyEvaluator(chatClientBuilder);
// Run the actual RAG flow from Phase 3's advanced RAG guide
Advisor policyRagAdvisor = RetrievalAugmentationAdvisor.builder()
.documentRetriever(VectorStoreDocumentRetriever.builder()
.vectorStore(vectorStore)
.similarityThreshold(0.60)
.build())
.build();
ChatResponse chatResponse = chatClient.prompt()
.advisors(policyRagAdvisor)
.user("What's the refund window for premium tier?")
.call()
.chatResponse();
EvaluationRequest request = new EvaluationRequest(
"What's the refund window for premium tier?",
chatResponse.getMetadata().get(RetrievalAugmentationAdvisor.DOCUMENT_CONTEXT),
chatResponse.getResult().getOutput().getText()
);
EvaluationResponse relevancy = relevancyEvaluator.evaluate(request);RelevancyEvaluator answers one narrow question: given the retrieved context, does the response actually address what was asked — not whether it's true, just whether it's on-topic and responsive. A bot that retrieves the right chunk but then rambles about something adjacent fails here even if nothing it said was factually wrong.
3. FactCheckingEvaluator: Is the Answer Actually Grounded?
This is the one that would have caught the regression from the top of this guide — it checks whether every claim in the response is actually supported by the retrieved context, catching hallucination even when the answer sounds relevant and confident.
FactCheckingEvaluator factCheckingEvaluator = new FactCheckingEvaluator(chatClientBuilder);
EvaluationResponse factCheck = factCheckingEvaluator.evaluate(request); // same EvaluationRequest as aboveRun both evaluators, not just one — they catch different failure modes. A response can be perfectly relevant (RelevancyEvaluator passes) while still inventing a detail the context never mentioned (FactCheckingEvaluator fails), and vice versa: a response can be fully grounded in the context yet fail to actually answer what the customer asked.
4. Beyond the Built-Ins: A Custom Rubric
RelevancyEvaluator and FactCheckingEvaluator cover grounding and relevance. They don't know your business rules — like the "never mention internal system names" constraint from this phase's Reflection guide. For anything domain-specific, write your own judge the same way: a ChatClient call, structured output, a pass/fail.
public record RubricResult(boolean passesReview, List<String> violations) {}
public RubricResult judgeAgainstRubric(String response) {
return judgeChatClient.prompt()
.user("""
Review this customer support response against these rules:
1. Must not mention internal system or database names.
2. Must not promise a specific refund amount without a tool-verified figure.
3. Must offer a next step if the answer is uncertain.
Response: %s
""".formatted(response))
.call()
.entity(RubricResult.class);
}This is the exact same LLM-as-a-judge idea RelevancyEvaluator and FactCheckingEvaluator are built from — a separate model call scoring output against explicit criteria — just pointed at a rubric only your domain cares about. Use a dedicated judgeChatClient here too, for the same reason the Reflection pattern used a separate reviewer: a model judging its own sibling response, with its own stricter system prompt, catches more than one judging itself.
5. Wiring It Into CI
The point of all three evaluators is running them automatically, not manually, against a small fixed set of representative questions — a "golden set" that should always produce grounded, relevant, on-policy answers:
@SpringBootTest
class SupportAgentEvaluationTest {
@Autowired ChatClient.Builder chatClientBuilder;
@Autowired VectorStore vectorStore;
record GoldenQuestion(String question, double minRelevancy) {}
static List<GoldenQuestion> goldenSet() {
return List.of(
new GoldenQuestion("What's the refund window for premium tier?", 0.9),
new GoldenQuestion("What's the refund window for joint accounts?", 0.9),
new GoldenQuestion("Can I get a refund after 120 days?", 0.9)
);
}
@ParameterizedTest
@MethodSource("goldenSet")
void ragAnswersAreRelevantAndGrounded(GoldenQuestion golden) {
ChatResponse response = buildChatClient().prompt()
.advisors(policyRagAdvisor())
.user(golden.question())
.call()
.chatResponse();
EvaluationRequest request = new EvaluationRequest(
golden.question(),
response.getMetadata().get(RetrievalAugmentationAdvisor.DOCUMENT_CONTEXT),
response.getResult().getOutput().getText()
);
assertThat(new RelevancyEvaluator(chatClientBuilder).evaluate(request).isPass())
.as("Relevancy: %s", golden.question())
.isTrue();
assertThat(new FactCheckingEvaluator(chatClientBuilder).evaluate(request).isPass())
.as("Fact-checking: %s", golden.question())
.isTrue();
}
}A prompt change, a model swap, or a policy document update that breaks grounding now fails a build instead of reaching a customer. Keep the golden set small and genuinely representative — ten well-chosen questions covering your actual edge cases catch more regressions than a hundred near-duplicates of the same easy question.
These tests make real model calls, which means real latency and real cost on every CI run. Run the full golden-set evaluation suite on a schedule or before a release, not on every single commit — reserve fast, free unit tests (mocking ChatModel) for that inner loop, and save the evaluator suite for the checkpoints where a regression slipping through actually matters.
Tracking these pass rates over time — not just pass/fail today, but the trend across the last twenty runs — is what actually catches slow drift instead of only hard breaks. That's a job for the observability stack this roadmap builds in a later phase: the same Micrometer pipeline that tracks token usage and latency is where evaluation scores belong too.
What's Next
Tool calling, hand-rolled agentic patterns, MCP interoperability, and now automated evaluation — this phase covered everything needed to build and trust a genuinely capable agent on plain Spring AI. The next phase introduces Embabel: a real planner for the cases this phase's "when to stop hand-rolling" section flagged, where the branching logic itself has outgrown a hand-written loop.
Frequently asked questions
Do I need a golden set of real production questions, or can I write synthetic ones?
Both, ideally — synthetic questions you write are good for covering known edge cases (the joint-account scenario, boundary dates like exactly 90 days), but real anonymized production questions catch phrasing and scenarios you wouldn't have thought to write yourself. Start synthetic, and add real questions as you observe actual failures worth guarding against permanently.
What's a reasonable threshold — should evaluators pass 100% of the time?
Treat isPass() as a boolean gate, not a tunable score — Spring AI's built-in evaluators already encode a YES/NO judgment in their prompt template, not a numeric threshold you set yourself. If you need a numeric score with a specific cutoff (like the roadmap's stretch goal of failing CI below 0.8), that's exactly the kind of rubric a custom LLM-as-a-judge evaluator, like the one in this guide, is for.
Can I evaluate tool-calling responses, not just RAG ones?
Yes — EvaluationRequest's context list doesn't have to come from a vector store. Populate it with whatever the response should have been grounded in, including a tool's returned data (like a RefundStatus object, serialized to text), and the same RelevancyEvaluator or a custom rubric evaluator works identically.
Should evaluation tests use the same model as production, or a stronger one?
A stronger or more careful judge model, when it's practical — the whole premise of LLM-as-a-judge is that judging is often an easier task than generating, but a judge that's weaker than the model it's grading will miss things a stronger one would catch. If cost rules out a stronger judge for every CI run, at least use one occasionally to sanity-check that your regular judge model isn't systematically too lenient.