The RAG Pipeline: Ingest, Chunk, Embed, Index, Retrieve
How the support bot's fabricated refund policy actually got fixed — the end-to-end pipeline that grounds a model's answers in real, citable documents.
The Fix, Finally
Three phases ago, a support bot confidently invented a "special 2024 refund policy" that never existed, and the mechanical explanation took a full phase to unpack: pre-training rewards plausible text, not verified text, and post-training rewards confident-sounding answers, not correct ones. Nothing in the model's training pipeline was ever going to fix that gap on its own — it needed a layer whose entire job is checking claims against a real source of truth. That layer is Retrieval-Augmented Generation (RAG), and it's the actual fix the team shipped: instead of trusting whatever the model learned during training, force every answer to be grounded in the company's real, current refund policy document, retrieved fresh on every question.
The term comes from Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020) — the paper that first combined a retriever (find relevant documents) with a generator (produce an answer conditioned on them) into one system, specifically to fight the knowledge-cutoff and fabrication problems that pure generation has no way to solve on its own.
The Five Stages
- Ingest — pull the raw source material: the actual refund policy document, pulled from wherever the company actually maintains it, not a snapshot baked into the model at training time.
- Chunk — split the document into smaller, retrievable units. A whole policy PDF is too much to usefully retrieve as one block; a single sentence is often too little context. Getting this right is its own discipline, covered in the next guide.
- Embed — convert each chunk into a vector using an embedding model, placing it in the latent space covered in Phase 1: chunks about similar topics end up near each other geometrically.
- Index — store those vectors in a structure built for fast similarity search (a vector database, or a vector index bolted onto an existing database) so retrieval doesn't mean scanning every chunk on every query.
- Retrieve — at query time, embed the user's question into that same vector space and pull back the chunks nearest to it — "nearby in the vector space" as a proxy for "relevant to the question," exactly as Phase 1 described.
None of this happens once and stays valid forever — a refund policy that changes next quarter needs re-ingesting, which is exactly the freshness problem the last guide in this phase covers.
IBM's What is RAG? is a solid plain-language overview of why this pattern exists at all, and LangChain's RAG tutorial walks through a working end-to-end implementation if you want to see the five stages as actual code.
Augment and Generate: Making the Model Show Its Work
Retrieval alone doesn't fix hallucination — it just finds the right text. The augment step is what actually closes the gap from Phase 1: the retrieved chunks get inserted into the model's context (Phase 3's context assembly, applied here specifically to retrieved content) alongside an instruction that changes the nature of the task entirely — not "answer this from what you know," but "answer this using only the following passages, and cite which passage supports each claim."
Answer the customer's question using ONLY the policy excerpts below.
If the excerpts don't contain the answer, say so explicitly — do not
guess. Cite the excerpt number for every claim you make.
[Excerpt 1] Refunds are processed within 5-7 business days for...
[Excerpt 2] Orders cancelled within 24 hours are eligible for...
Customer question: How long does a refund take?
This is what the roadmap's own fix describes as "Strict Grounding" — forcing citations doesn't make the model incapable of fabricating (nothing does, fully), but it changes what a fabrication looks like: instead of a fluent, unverifiable claim, a hallucinated citation is now a checkable one. A guardrail layer — covered in Phase 9 — can mechanically verify that Excerpt 2 actually says what the model claims it says, which was never possible against the model's untraceable training-time "knowledge." That single structural change is most of why the hallucination rate on the refund bot dropped by 95%.
Check yourself
Why does forcing the model to cite a specific retrieved excerpt for each claim reduce hallucination more effectively than just asking it to 'be accurate'?
What's Next
The pipeline above glossed over one decision that determines whether retrieval actually finds the right thing: how a document gets split into chunks in the first place, and which embedding model turns those chunks into vectors worth searching. That's next.
Frequently asked questions
Does RAG eliminate hallucination completely?
No — it reduces it substantially by giving the model something to ground answers in and a citation structure a guardrail can verify, but a model can still misread or misrepresent a retrieved passage. The roadmap's own numbers for this exact scenario are a 95% reduction, not 100% — RAG is the biggest single lever, not a complete solve, which is why Phase 9's guardrails and evaluation layer still matter on top of it.
Is RAG the same thing as fine-tuning a model on your documents?
No, and they solve different problems. Fine-tuning changes the model's weights based on training examples, which is expensive, slow to update, and still doesn't give you a checkable citation for any specific claim. RAG keeps the model unchanged and instead controls what's in its context at query time, which is cheap to update (re-index a changed document) and produces claims traceable to a specific source.
Do I need a dedicated vector database, or can I bolt vector search onto an existing database?
Either can work — dedicated vector databases (Pinecone, Weaviate, Qdrant, and others) are optimized specifically for large-scale similarity search, while several general-purpose databases (Postgres with pgvector, for example) now support vector columns well enough for moderate scale. The right choice depends on your document volume and query latency requirements more than any inherent superiority of one approach.