RAG Part 2: When Similarity Search Isn't Enough

Three More Complaints

The policy Q&A bot from the last guide shipped and mostly worked — "mostly" being the operative word once real customers started typing real questions instead of the clean, well-formed ones used to demo it:

1. "I asked 'what about the 90 day thing' and got nothing useful." The customer had refund policy in mind, but the words "90 day thing" barely overlap, semantically, with anything in the actual policy document. QuestionAnswerAdvisor searched for exactly what was typed and found a mediocre match.
2. "I asked about premium refunds, then asked 'and what about joint accounts?' — it searched for joint accounts in general, not joint accounts and refunds." ChatMemory (from guide one of this phase's predecessor) correctly resent the conversation history to the model — but the vector search itself only ever embedded the literal follow-up text, with no idea it was a continuation of anything.
3. "A customer wrote their question in Spanish and got an answer that didn't match their question at all." The embedding model was trained predominantly on English text; a Spanish query and an English policy chunk don't land as close together in vector space as you'd hope.

QuestionAnswerAdvisor does exactly one thing: embed the literal query, run one similarity search, stuff the results into context. All three complaints above are really the same lesson — the query you search with is not automatically the query you should search with — and Spring AI's modular RAG API exists specifically to fix that, one composable piece at a time.

1. RetrievalAugmentationAdvisor: RAG as Four Swappable Stages

// build.gradle
implementation 'org.springframework.ai:spring-ai-rag'

RetrievalAugmentationAdvisor replaces QuestionAnswerAdvisor's one step with four, each independently swappable:

The simplest possible RetrievalAugmentationAdvisor — just the retrieval stage, everything else defaulted — behaves almost exactly like QuestionAnswerAdvisor:

Advisor advisor = RetrievalAugmentationAdvisor.builder()
    .documentRetriever(VectorStoreDocumentRetriever.builder()
        .vectorStore(vectorStore)
        .similarityThreshold(0.50)
        .build())
    .build();

The rest of this guide fills in the other three stages, one complaint at a time.

2. Pre-Retrieval: Fixing the Query Before It's Searched

Complaint #1: vague queries — RewriteQueryTransformer

RewriteQueryTransformer asks the model itself to rewrite a vague or verbose query into something that retrieves better, before it ever reaches the vector store:

ChatClient.Builder lowTempBuilder = chatClientBuilder.defaultOptions(
    ChatOptions.builder().temperature(0.0).build());
 
QueryTransformer rewriter = RewriteQueryTransformer.builder()
    .chatClientBuilder(lowTempBuilder)
    .build();
 
Query rewritten = rewriter.transform(new Query("what about the 90 day thing"));
// rewritten.text() might be "What is the 90-day policy?" — close enough
// to the actual document wording to retrieve the right chunk.

Complaint #2: follow-up questions — CompressionQueryTransformer

This is the one that fixes "and what about joint accounts?" CompressionQueryTransformer folds conversation history and a follow-up into a single, standalone query — solving a different problem than ChatMemory solves. Memory makes sure the model sees prior turns; this makes sure the retrieval step searches for what the follow-up actually means.

Query query = Query.builder()
    .text("and what about joint accounts?")
    .history(
        new UserMessage("What's the refund window for premium tier?"),
        new AssistantMessage("Premium tier customers have a 90-day refund window.")
    )
    .build();
 
QueryTransformer compressor = CompressionQueryTransformer.builder()
    .chatClientBuilder(lowTempBuilder)
    .build();
 
Query standalone = compressor.transform(query);
// standalone.text() ≈ "What is the refund window for joint accounts?"

Complaint #3: non-English input — TranslationQueryTransformer

QueryTransformer translator = TranslationQueryTransformer.builder()
    .chatClientBuilder(lowTempBuilder)
    .targetLanguage("english")
    .build();
 
Query translated = translator.transform(new Query("¿Cuál es la política de reembolso?"));

Translate the query to match the language your documents (and embedding model) were trained on — the answer still comes back in whatever language the model's response is generated in, which you control separately with the system prompt.

Searching multiple angles at once: MultiQueryExpander

Instead of picking one rewritten query, MultiQueryExpander generates several semantically diverse variations and searches with all of them — useful when a single rewrite might still miss the phrasing your documents actually use:

MultiQueryExpander expander = MultiQueryExpander.builder()
    .chatClientBuilder(lowTempBuilder)
    .numberOfQueries(3)
    .includeOriginal(true)
    .build();
 
List<Query> queries = expander.expand(new Query("what about the 90 day thing"));
// 4 queries in total: the original, plus 3 model-generated variations

RetrievalAugmentationAdvisor runs retrieval once per expanded query and merges the results with a DocumentJoiner (ConcatenationDocumentJoiner deduplicates while keeping the first occurrence of each chunk) — you rarely call .expand() directly; .queryTransformers() on the builder wires this in for you, shown in the combined example below.

3. Retrieval: Filtering by Metadata, Properly

Guide one stamped tier: "premium" onto each chunk and left filtering as an FAQ answer. Here's the actual code, using VectorStoreDocumentRetriever instead of calling vectorStore.similaritySearch() directly:

DocumentRetriever retriever = VectorStoreDocumentRetriever.builder()
    .vectorStore(vectorStore)
    .similarityThreshold(0.60)
    .topK(5)
    .filterExpression(new FilterExpressionBuilder()
        .eq("tier", customerTier)
        .build())
    .build();

A lower-tier customer's query now never even considers premium-only chunks as candidates — the filter runs inside the similarity search, not as a check on the results afterward.

4. Post-Retrieval: Re-Ranking Is Bring-Your-Own

DocumentPostProcessor is the hook for cleaning up retrieved chunks before they reach the model — trimming redundant ones, fighting the "lost in the middle" effect on long context, or re-ranking by a signal better than raw cosine similarity. Spring AI defines the interface; it does not ship a built-in cross-encoder reranker, so this stage is either skipped, hand-rolled, or backed by a third-party re-ranking API:

public class RecencyBoostPostProcessor implements DocumentPostProcessor {
    @Override
    public List<Document> process(Query query, List<Document> documents) {
        return documents.stream()
            .sorted(Comparator.comparing(
                d -> (String) d.getMetadata().getOrDefault("updatedAt", "1970-01-01"),
                Comparator.reverseOrder()))
            .toList();
    }
}

That's a deliberately simple example — sort retrieved chunks so the most recently updated policy wins ties — not a real semantic reranker. If retrieval quality genuinely needs cross-encoder re-ranking, that's the point where you'd integrate a dedicated reranking model or API behind this same interface, rather than Spring AI providing one out of the box.

5. Generation: Enforcing "Don't Guess" Structurally

Guide one's fix for hallucination was a system prompt instruction: "if the context doesn't contain the answer, say you don't have that information." That's a request, not a guarantee — the model can still ignore it. ContextualQueryAugmenter gives you a structural version of the same rule:

QueryAugmenter augmenter = ContextualQueryAugmenter.builder()
    .allowEmptyContext(false) // default — rejects generation when retrieval found nothing
    .build();

With allowEmptyContext(false), an empty retrieval result short-circuits before the model is even asked to generate a free-form guess. Flip it to true only for use cases where some answer, even an ungrounded one, beats no answer at all — that's rarely true for a policy bot, and rarely a good default.

6. Putting It Together

A RetrievalAugmentationAdvisor addressing all three original complaints — vague-query rewriting, multi-query fan-out, tier-based filtering, and a hard "don't guess" rule — composed once and reused like any other advisor:

Advisor policyRagAdvisor = RetrievalAugmentationAdvisor.builder()
    .queryTransformers(
        RewriteQueryTransformer.builder().chatClientBuilder(lowTempBuilder).build()
    )
    .documentRetriever(VectorStoreDocumentRetriever.builder()
        .vectorStore(vectorStore)
        .similarityThreshold(0.60)
        .topK(5)
        .filterExpression(new FilterExpressionBuilder().eq("tier", customerTier).build())
        .build())
    .queryAugmenter(ContextualQueryAugmenter.builder()
        .allowEmptyContext(false)
        .build())
    .build();
 
String answer = chatClient.prompt()
    .advisors(policyRagAdvisor)
    .user(question)
    .call()
    .content();

Same ChatClient, same calling convention as QuestionAnswerAdvisor from the last guide — only the advisor passed to .advisors() changed. That's the entire point of the four-stage design: none of the surrounding application code needs to know or care how retrieval got smarter.

What's Next

RAG is done for this roadmap — grounded, filtered, and now resilient to the messy queries real customers actually type. The next guide moves from answering questions to acting on them: tool calling, MCP interoperability, and evaluating whether an agent's answers hold up.