Chunking & Embedding Strategy
Why splitting a policy document into fixed 500-character blocks quietly broke retrieval, and how chunk size, overlap, and embedding model choice fix it.
The Retrieval That Almost Worked
The first version of the refund bot's RAG pipeline split the policy document into fixed 500-character blocks and called it done. Retrieval quality was mediocre in a specific, confusing way: the right document almost always got retrieved, but sometimes the specific sentence answering the question landed in a different chunk than the one retrieved, because the split point fell in the middle of the relevant paragraph. The pipeline wasn't broken — the chunking was working against it.
Three Ways to Split a Document
Fixed-size chunking splits text into blocks of a set length (characters or tokens), regardless of what's actually in them. It's simple and predictable, and it's exactly what caused the refund bot's problem: a 500-character boundary has no concept of "sentence" or "paragraph," so it will cut a policy clause in half as readily as it cuts between two unrelated ones.
Sliding window chunking fixes the worst of that by overlapping consecutive chunks — chunk 2 repeats the last portion of chunk 1 — so a sentence split across a boundary is very likely to appear whole in at least one of the two overlapping chunks. The cost is straightforward: some content gets embedded and stored more than once, which means more vectors to search and a bit more storage, in exchange for far fewer split-sentence failures.
Semantic chunking splits at meaning boundaries instead of a fixed character count — section headers, paragraph breaks, or points where consecutive sentences' embeddings diverge enough to suggest a topic shift. This is the most expensive to compute (it requires at least a pass of embedding or structural analysis before you even know where the chunks are) and generally the most retrieval-friendly, because chunks correspond to coherent ideas rather than arbitrary character counts.
| Strategy | How it splits | Best for | Cost |
|---|---|---|---|
| Fixed-size | Every N characters/tokens | Simple, uniform text (logs, transcripts) | Lowest to compute |
| Sliding window | Fixed size, with overlap | General documents where boundary splits matter | Some storage/index duplication |
| Semantic | Meaning boundaries | Structured documents with real topical sections | Highest to compute |
Pinecone's guide to document chunking strategies is a good practical reference for implementation details across all three approaches, including how to pick a chunk size for a specific document type.
The Size Tradeoff Nobody's Immune To
Every chunking strategy still has to answer "how big should a chunk actually be," and there's no universal right answer, only a real tradeoff:
- Too small, and a chunk loses the surrounding context that gives it meaning — a chunk containing only "5-7 business days" with no indication of what takes 5-7 business days is nearly useless once retrieved on its own.
- Too large, and a chunk's embedding becomes a blurred average of everything in it — a 3,000-character chunk covering refunds, exchanges, and warranty claims embeds somewhere in the middle of all three topics, making it a mediocre match for a query about any single one of them, and wastes context budget (Phase 3) once retrieved.
The refund bot's team settled on roughly paragraph-sized chunks with a modest overlap — small enough that each chunk stays on one topic, large enough that a chunk read alone still makes sense, with overlap catching the sentences fixed-size splitting alone would have cut.
Choosing an Embedding Model
The chunking strategy decides what gets embedded; the embedding model decides how well "nearby in vector space" actually tracks "relevant in meaning." A few practical factors that matter more than chasing the top of a leaderboard:
- Domain fit. A general-purpose embedding model trained mostly on web text handles a refund policy document fine; a codebase or a legal-contract corpus benefits from models specifically evaluated on that kind of text.
- Dimensionality and cost. Higher-dimensional embeddings can capture more nuance but cost more to store and search at scale — for a few hundred policy chunks this is irrelevant; for millions of documents it becomes a real infrastructure decision, one Phase 5 covers in depth.
- Consistency. As covered in Phase 1, embeddings from different models aren't comparable — switching embedding models means re-embedding your entire corpus, not just new documents going forward.
MTEB (Massive Text Embedding Benchmark) is the standard reference for comparing embedding models across a broad range of tasks (retrieval, classification, clustering) rather than trusting any single vendor's claims — worth checking before committing to a model, since performance varies meaningfully by task type, not just by overall score.
Check yourself
A RAG pipeline retrieves the right document almost every time, but the specific answer to the user's question often ends up split across the boundary between two chunks. What's the most likely fix?
What's Next
Chunking and embedding strategy gets retrieval working well for a single, static knowledge base. Production systems rarely stay that simple — the refund bot's knowledge base grows to include internal-only documents, the policy changes quarterly, and eventually the same index needs to serve multiple customers without leaking each other's data. That's next.
Frequently asked questions
Is there a single 'best' chunk size that works across document types?
No — it depends on the document's natural structure and the kind of question users ask. A FAQ document with short, self-contained answers wants smaller chunks than a legal contract where clauses reference earlier definitions. Testing retrieval quality against real user questions, rather than picking a size from a blog post, is the only reliable way to land on the right value for a specific corpus.
Should overlap always be added to fixed-size chunking?
It's a reasonable default for prose documents, but it isn't free — every overlapping token is stored and searched twice. For text with hard natural breaks (log lines, individual FAQ entries, code functions) where a split rarely lands mid-thought, the overlap cost may not buy much and can be skipped.
Can I mix chunking strategies in one knowledge base?
Yes, and it's common in practice — a knowledge base combining a structured FAQ and a long-form policy PDF often benefits from semantic chunking on the FAQ (splitting by question) and sliding-window chunking on the PDF (where topic boundaries are less clearly marked). The chunking strategy is a property of the document, not a global setting for the whole pipeline.