Grounding Answers in Your Own Data: Embeddings, Vector Stores & RAG

The Confidently Wrong Refund Policy

Three weeks after the memory-enabled support bot from the last guide went live, an escalation lands on the fintech team's desk: the bot told a customer their premium-tier refund window was "30 days from purchase." Acme Fintech's actual policy, updated two months ago, gives premium-tier customers a 90-day window. The bot wasn't lying — it simply answered from whatever it learned during training, which has no idea Acme Fintech exists, let alone what its refund policy says today.

This is the gap every guide so far has quietly left open. ChatClient, PromptTemplate, ChatMemory — none of them give the model access to facts outside its training data. That's what Retrieval-Augmented Generation (RAG) is for: look up the relevant part of your documents first, then hand it to the model as context, so it answers from what's actually true today instead of what it remembers from training.

1. Embeddings: Turning Text Into Something Searchable

You can't run a SQL WHERE clause against "does this paragraph talk about refunds?" — but you can compare meaning. An embedding model converts text into a dense vector of floating-point numbers (1536 of them, for OpenAI's text-embedding-3-small) positioned in space such that semantically similar text ends up as nearby vectors, regardless of the exact words used. "Refund window" and "how long do I have to get my money back" land close together; "refund window" and "password reset" don't.

@Autowired
EmbeddingModel embeddingModel;
 
float[] vector = embeddingModel.embed("What is the refund window for premium customers?");
// vector.length == 1536 for OpenAI's text-embedding-3-small

You'll rarely call .embed() directly like this — VectorStore (next section) does it for you on the way in and on the way out. The one-liner above just makes the abstraction concrete: an embedding is a coordinate, and "search" becomes "find the nearest coordinates," typically measured with cosine similarity.

2. Ingesting Documents: Read, Then Chunk

RAG starts with getting your documents into the pipeline. Spring AI's DocumentReader implementations handle the parsing; TextSplitter implementations handle breaking the result into chunks small enough to embed and retrieve individually — you don't want "the entire 40-page policy PDF" as a single search result.

// build.gradle
implementation 'org.springframework.ai:spring-ai-pdf-document-reader'

PagePdfDocumentReader pdfReader = new PagePdfDocumentReader(
    "classpath:/policies/refund-policy.pdf",
    PdfDocumentReaderConfig.builder()
        .withPagesPerDocument(1)
        .build()
);
List<Document> pages = pdfReader.read();
 
TokenTextSplitter splitter = TokenTextSplitter.builder()
    .withChunkSize(800)
    .withMinChunkSizeChars(350)
    .build();
List<Document> chunks = splitter.apply(pages);

Each Document carries its text plus a Map<String, Object> of metadata. Stamp useful metadata on the way in — it's what lets you filter or cite results later:

chunks.forEach(chunk -> chunk.getMetadata().putAll(Map.of(
    "source", "refund-policy.pdf",
    "tier", "premium",
    "updatedAt", "2026-06-01"
)));

3. Vector Stores: Where the Chunks Live

A VectorStore embeds each Document and stores the vector alongside the original text and metadata, then answers "give me the K chunks closest to this query" on demand. Spring AI's VectorStore interface is the same shape regardless of backend — Chroma for a quick local Docker container, Pinecone if you want fully managed, or PGVector if you already run Postgres and would rather not add a new piece of infrastructure just for this.

// build.gradle
implementation 'org.springframework.ai:spring-ai-pgvector-store-spring-boot-starter'
implementation 'org.springframework.boot:spring-boot-starter-data-jdbc'

# application.yml
spring:
  ai:
    vectorstore:
      pgvector:
        initialize-schema: true
  datasource:
    url: jdbc:postgresql://localhost:5432/acme
    username: acme
    password: ${DB_PASSWORD}

Loading and querying is two method calls against the same auto-configured bean:

@Autowired
VectorStore vectorStore;
 
void loadRefundPolicy(List<Document> chunks) {
    vectorStore.add(chunks);
}
 
List<Document> findRelevant(String question) {
    SearchRequest request = SearchRequest.builder()
        .query(question)
        .topK(5)
        .similarityThreshold(0.7)
        .build();
    return vectorStore.similaritySearch(request);
}

similarityThreshold(0.7) matters as much as topK(5) — without it, a topK search always returns something, even if the closest chunk is a poor match for a question about a topic your documents don't cover at all. A threshold gives the retriever permission to come back with nothing rather than force-feeding the model an irrelevant chunk.

4. RAG: Wiring Retrieval Into ChatClient

Recall from the last guide: advisors are how ChatClient grows new capabilities without changing your method signatures. RAG is just another advisor — QuestionAnswerAdvisor retrieves relevant chunks from a VectorStore and inserts them into the prompt as context, automatically, on every call it's attached to.

@Service
public class PolicyQaService {
 
    private final ChatClient chatClient;
 
    public PolicyQaService(ChatClient.Builder builder, VectorStore vectorStore) {
        this.chatClient = builder
            .defaultSystem("""
                You answer questions about Acme Fintech's policies.
                Only answer using the provided context. If the context doesn't
                contain the answer, say you don't have that information —
                never guess.
                """)
            .defaultAdvisors(QuestionAnswerAdvisor.builder(vectorStore).build())
            .build();
    }
 
    public String ask(String question) {
        return chatClient.prompt()
            .user(question)
            .call()
            .content();
    }
}

That system prompt instruction — "if the context doesn't contain the answer, say you don't have that information" — is doing real work. QuestionAnswerAdvisor gives the model the opportunity to answer correctly; it doesn't stop the model from ignoring the context and answering from training data anyway. Telling it explicitly not to guess is what actually closes the escalation from the top of this guide.

Combining memory and RAG on the same ChatClient

Advisors compose. Attach both, and the bot remembers the conversation and answers from your documents in the same call:

this.chatClient = builder
    .defaultSystem("Answer Acme Fintech policy questions using only the provided context.")
    .defaultAdvisors(
        MessageChatMemoryAdvisor.builder(chatMemory).build(),
        QuestionAnswerAdvisor.builder(vectorStore).build()
    )
    .build();

A customer can now ask "what's the refund window for my tier?", get a grounded answer from the actual policy document, then follow up with "and what about for a joint account?" — memory carries the first question's context, RAG re-retrieves fresh chunks for the follow-up.

5. Putting It Together

@Service
public class PolicyQaService {
 
    private final ChatClient chatClient;
    private final VectorStore vectorStore;
 
    public PolicyQaService(ChatClient.Builder builder, VectorStore vectorStore, ChatMemory chatMemory) {
        this.vectorStore = vectorStore;
        this.chatClient = builder
            .defaultSystem("""
                You answer questions about Acme Fintech's policies for customer tier {tier}.
                Only answer using the provided context. If the context doesn't
                contain the answer, say you don't have that information — never guess.
                """)
            .defaultAdvisors(
                MessageChatMemoryAdvisor.builder(chatMemory).build(),
                QuestionAnswerAdvisor.builder(vectorStore).build()
            )
            .build();
    }
 
    public void ingest(List<Document> chunks) {
        vectorStore.add(chunks);
    }
 
    public String ask(String conversationId, String question) {
        return chatClient.prompt()
            .user(question)
            .advisors(a -> a.param(ChatMemory.CONVERSATION_ID, conversationId))
            .call()
            .content();
    }
}

One ingestion path for getting policy documents in, one ask() method that's simultaneously stateful (memory) and grounded (RAG) — and neither advisor knows the other one exists.

What's Next

The bot now answers from your own documents instead of guessing. The next guide gives it the ability to act, not just answer — calling real Java methods and REST APIs mid-conversation with tool calling, interoperating with other agents over MCP, and evaluating whether the answers it gives (retrieved or otherwise) actually hold up.