The Core Idea: A document is too long to embed, search and place in a prompt as one piece, so it is cut into chunks. A chunk is the unit that RAG indexes, retrieves and shows to the model. Cut too small and the facts lose their context. Cut too large and each chunk mixes many topics and matches nothing precisely. Good chunking is one of the biggest levers on RAG quality.
1. Beginner: Why Documents Must Be Chunked
Imagine searching a 300-page manual by asking a librarian for the whole book every time. You would rather ask for the exact page. Chunking turns a long document into page-sized pieces so retrieval can return just the part that answers the question.
Embedding limits
Embedding models accept only a limited amount of text, and one vector for a whole book blurs many topics into a single average meaning.
Precision
A small focused chunk matches a specific question strongly. A huge chunk matches weakly because most of it is unrelated.
Prompt budget
The prompt can hold only a few passages. Short relevant chunks fit many sources; long ones crowd each other out.
Key point: the chunk is the unit of retrieval. Whatever you cut at indexing time is exactly what the retriever can find and the model can read later.
2. Beginner: Fixed-Size Chunking and Overlap
The simplest method cuts the text every N tokens or words, ignoring its meaning. It is fast, predictable and easy to tune. Its weakness is that a cut can land in the middle of a fact, leaving half in one chunk and half in the next.
Overlap repeats the last few words of one chunk at the start of the next, so a fact that crosses a boundary still appears whole in at least one chunk. With text length , chunk size and overlap (all in the same unit), the number of chunks is about:
For example, , and gives chunks. Overlap costs extra storage and embedding work, so teams often start with a modest overlap, commonly around 10 to 20 percent of the chunk size, and tune it.
3. Intermediate: Sentence, Paragraph and Recursive Chunking
Structure-aware methods cut along natural boundaries instead of a fixed count, so chunks read like complete thoughts.
| Strategy | How it cuts | Strength | Weakness |
|---|---|---|---|
| Sentence | Groups a few whole sentences | Never splits a sentence | Sizes vary a lot |
| Paragraph | One chunk per paragraph | Keeps a full idea together | Paragraphs can be tiny or huge |
| Recursive | Tries big separators first, then smaller ones until each piece fits | Good default; respects structure and size limit | Needs sensible separators |
How recursive chunking works: split on paragraph breaks first. Any piece still too long is split on line breaks, then sentences, then words. Each piece is kept as large as the size limit allows, so structure is preserved wherever possible.
4. Intermediate: Semantic Chunking
Semantic chunking starts a new chunk when the topic changes. The text is split into sentences, each is embedded, and the similarity between neighboring sentences is measured. Where similarity drops below a threshold, a boundary is placed.
Step 1. Split the document into sentences.
Step 2. Embed every sentence.
Step 3. Compare each sentence with the next one.
Step 4. Cut where the similarity falls sharply, then merge small leftovers.
Trade-off: semantic chunks follow meaning, but they need an embedding call for every sentence at indexing time and a threshold that must be tuned. Many teams find recursive chunking nearly as good and far cheaper, so measure before you commit.
5. Advanced: Choosing a Chunk Size by Measuring
There is no universal best size. Small chunks give precise matches but may lack context. Large chunks carry context but dilute the match and use more of the prompt budget.
| Aspect | Small chunks | Large chunks |
|---|---|---|
| Match precision | High | Lower |
| Context inside the chunk | Often too little | Usually enough |
| Prompt space used | Little each | Much each |
| Index size | More vectors | Fewer vectors |
A common starting point is a few hundred tokens per chunk with light overlap, but treat that as a hypothesis. Build a small set of real questions with known answer locations, try several sizes, and compare how often the right chunk appears in the top results. The number that wins on your data is the right one.
Always keep metadata: store the source, section and position of every chunk. It makes citations possible, lets you fetch neighboring chunks, and prepares the ground for the parent and child chunks in the next lesson.
6. Visual Explanation: Cutting a Document Four Ways
The figure has three panels: the chunking flow, a comparison of small and large chunks, and an explorer for the four main strategies.
Panel A · The Chunking FlowEvery strategy follows the same path
Clean document→Choose a cutting rule→Add overlap→Chunks with metadata
Panel B · Small vs Large ChunksNeither is always better
Small chunks
About a sentence or two
Very focused
↓
Strong match to a precise question
↓
Risk: missing context
Large chunks
Several paragraphs
Rich context
↓
Mixed topics dilute the match
↓
Risk: noisy, uses prompt space
Panel C · Strategy ExplorerClick a strategy to open it
Click to see why overlap helps▼
If a key fact sits across a cut, neither chunk contains it whole. Repeating a few words at the boundary makes sure at least one chunk holds the full fact.
How to read it: Panel A is the common path, Panel B shows the central trade-off, and Panel C helps you pick a starting strategy.
7. Python Implementation: Three Chunkers Compared
The program cuts one short policy three ways: fixed-size without overlap, fixed-size with overlap, and one sentence per chunk. It then checks which strategies keep the phrase payment method within 5 business days together in a single chunk, which is the fact a user might ask about.
chunkers.pyPython 3.10+ · Standard library only
text = ("Returns are accepted within 30 days of purchase. " "The item must be unused and in its original box. " "Refunds go to the original payment method within 5 business days. " "Gift cards cannot be refunded.") def fixed_chunks(text, size, overlap): words = text.split() step = size - overlap chunks = [] start = 0 while start < len(words): chunks.append(" ".join(words[start:start + size])) if start + size >= len(words): break start += step return chunks def sentence_chunks(text, per_chunk): sentences = [s.strip() + "." for s in text.split(".") if s.strip()] chunks = [] for i in range(0, len(sentences), per_chunk): chunks.append(" ".join(sentences[i:i + per_chunk])) return chunks def show(title, chunks): print(title, "-", len(chunks), "chunks") for i, chunk in enumerate(chunks): print(" " + str(i + 1) + ".", chunk) target = "payment method within 5 business days"no_overlap = fixed_chunks(text, 13, 0)with_overlap = fixed_chunks(text, 13, 4)by_sentence = sentence_chunks(text, 1) show("Fixed, no overlap", no_overlap)show("Fixed, overlap 4", with_overlap)show("Sentence", by_sentence)print("")for name, chunks in [("Fixed, no overlap", no_overlap), ("Fixed, overlap 4", with_overlap), ("Sentence", by_sentence)]: held = any(target in chunk for chunk in chunks) print(name.ljust(18), "keeps the fact whole:", held)Expected output
Fixed, no overlap - 3 chunks 1. Returns are accepted within 30 days of purchase. The item must be unused 2. and in its original box. Refunds go to the original payment method within 3. 5 business days. Gift cards cannot be refunded.Fixed, overlap 4 - 4 chunks 1. Returns are accepted within 30 days of purchase. The item must be unused 2. item must be unused and in its original box. Refunds go to the 3. Refunds go to the original payment method within 5 business days. Gift cards 4. business days. Gift cards cannot be refunded.Sentence - 4 chunks 1. Returns are accepted within 30 days of purchase. 2. The item must be unused and in its original box. 3. Refunds go to the original payment method within 5 business days. 4. Gift cards cannot be refunded. Fixed, no overlap keeps the fact whole: FalseFixed, overlap 4 keeps the fact whole: TrueSentence keeps the fact whole: TrueRead the result line by line. Cutting every 13 words without overlap splits the key phrase across two chunks, so no single chunk holds the whole fact. Adding an overlap of 4 words, or cutting at sentence boundaries, repairs that. The overlap here is large compared with the chunk size only because the example text is tiny. The exact outcome always depends on where the cuts fall, which is why you should test chunking on your own documents.
Knowledge Check
Q1. Why is overlap added between fixed-size chunks?
Q2. A teammate sets very large chunks. What is the main risk?
Q3. What does recursive chunking try first?
Key Points
✓
Documents are chunked because embeddings, retrieval precision and the prompt budget all work better with focused pieces.
✓
Fixed-size chunking is simple but can cut through facts; overlap reduces that risk.
✓
Sentence, paragraph and recursive chunking follow structure and read more naturally.
✓
Semantic chunking cuts where meaning shifts, at the price of extra embedding cost and tuning.
✓
Small chunks match precisely but lose context; large chunks keep context but blur the match.
✓
Choose chunk size by testing retrieval on real questions, not by rule of thumb alone.
✓
Keep source, section and position metadata for every chunk.
The Big Picture
Chunking decides what the retriever is even able to find. Every later stage, from embeddings to reranking, works on the pieces you create here, so it pays to inspect your chunks with your own eyes and to measure alternatives.
Clean text → Choose a cutting rule → Add overlap → Attach metadata → Index
Up Next: Advanced Chunking and Document Structure. You will learn parent and child chunks, hierarchical chunks, structure-aware splitting by headers, and how to handle tables and relationships between sections.