Advanced Chunking and Document Structure

Learn parent-child and hierarchical chunks, structure-aware splitting by headers, how to handle tables and lists, and how to keep relationships between chunks.

25 minIntermediateCode Examples
The Core Idea: Plain chunking treats a document as a flat stream of words. Real documents have structure: titles, sections, lists, tables and links between parts. Advanced chunking uses that structure so the retriever can find small precise pieces while the model still receives enough surrounding context to answer correctly.

1. Intermediate: The Small-to-Big Problem

Retrieval and generation want opposite things. Retrieval works best with small, focused chunks, because a short chunk produces a sharp embedding. Generation works best with larger passages, because the model needs surrounding sentences to answer fully and correctly.
The fix is to separate the unit you search from the unit you read: search over small pieces, then hand the model a larger piece around the match. This idea is often called small-to-big retrieval.
Search small
Index short chunks, such as single sentences or short paragraphs, for precise matching.
Read big
After a match, expand to the surrounding paragraph or section before building the prompt.
Link them
Store an id on every small chunk that points to its larger parent so the expansion is a quick lookup.

2. Intermediate: Parent-Child and Hierarchical Chunks

In parent-child chunking every small child chunk keeps the id of the larger parent chunk it was cut from. Only children are embedded and searched. When a child matches, the system returns its parent.
Hierarchical chunking extends this to many levels: document, section, paragraph, sentence. Matches can be expanded one level or several, depending on how much context the question needs.
Expansion methodWhat the model receivesBest when
Parent chunkThe whole parent of the matching childDocuments have clear sections
Sentence windowThe matching sentence plus a few neighbors on each sideDense prose without headings
Auto-mergeThe parent, but only if many of its children matchedQuestions that span a whole section
Watch the budget: bigger context can bury the answer and use up prompt space. Expand only as far as needed, and merge duplicates when several children share one parent.

3. Intermediate: Structure-Aware Chunking

Many documents announce their own structure through headings: Markdown headers, HTML heading tags, numbered sections in PDFs. Structure-aware chunking splits at those boundaries so a chunk never mixes two sections.
A powerful trick is to keep the heading path with each chunk, for example Returns, then Refunds. The path can be stored as metadata and also placed at the start of the chunk text. A sentence like They take 5 business days becomes self-explanatory once it begins with the heading it belongs to, and the embedding captures that topic too.
Rule of thumb: split on the biggest structural boundary available first, and fall back to size-based splitting only for sections that are still too long.

4. Intermediate: Tables, Lists and Special Content

Content that is not ordinary prose needs its own rules, because cutting it carelessly destroys its meaning.
Tables
Never cut a row in half. For small tables keep the whole table in one chunk. For large ones turn each row into a line that repeats the column names, such as Plan Basic, Price 10.
Lists
Keep a list together with the sentence that introduces it. A bullet list without its lead-in often has no meaning.
Code and figures
Keep code blocks whole and attach captions or alt text to figures, so the surrounding explanation travels with them.
Check by reading: print a few table and list chunks and ask whether a person could answer a question from that chunk alone. If not, the chunking needs more context.

5. Advanced: Relationships Between Chunks

Chunks are not independent. Sections refer to each other, a summary points at the details, and versions replace older ones. Recording these links lets the retriever follow them.
LinkStored asUsed for
ParentA parent id on each childSmall-to-big expansion
NeighborsPrevious and next chunk idsSentence windows and reading on
ReferencesIds of sections that are mentioned, such as see section 4Following citations inside documents
VersionsDocument id plus version and statusHiding outdated chunks
Each extra link adds indexing work and more ways for the system to go wrong, for example expanding too far and flooding the prompt. Add relationships when evaluation shows that single chunks are missing context, not before.

6. Visual Explanation: Search Small, Read Big

Panel A shows the hierarchy of a document. Panel B contrasts the unit that is searched with the unit that is read. Panel C is an explorer of special content types.
Panel A · The HierarchyEach level can be a chunk size
Document→Sections→Paragraphs→Sentences
Search at the lowest level, read at a higher level
Panel B · Child Searched, Parent ReadTwo jobs, two sizes
Child chunk
Used for search
Short and focused
↓
Sharp embedding
↓
Finds the exact match
Parent chunk
Used for reading
Whole section with heading
↓
Full context
↓
Goes into the prompt
Panel C · Special ContentClick a type to open it
Click to see the cost of bigger context▼
Returning a parent is safer for answers, but it uses more prompt space and can include unrelated text. Measure whether the extra context really improves answers before expanding everywhere.
How to read it: Panel A names the levels, Panel B assigns each a job, and Panel C lists the traps that special content sets for a careless splitter.

7. Python Implementation: Header Paths, Parent-Child Search and Table Rows

The program splits a small Markdown document into sections and keeps the heading path. Every sentence becomes a child chunk that remembers its parent section. A search matches the best child, then returns the whole parent section for the model to read. The last lines turn table rows into self-contained chunks that repeat their column names.
structure_chunking.pyPython 3.10+ · Standard library only
doc_lines = [    "# Returns",    "## Refunds",    "Refunds go to the original payment method. They take 5 business days.",    "## Exchanges",    "Exchanges are free within 30 days. Items must be unused.",    "# Shipping",    "## Delivery",    "Standard delivery takes 5 days. Express delivery takes 2 days.",] def words(text):    cleaned = "".join(ch for ch in text.lower() if ch.isalnum() or ch == " ")    return set(cleaned.split()) def split_sections(lines):    sections = []    h1 = ""    h2 = ""    for line in lines:        if line.startswith("## "):            h2 = line[3:]            sections.append([h1 + " > " + h2, ""])        elif line.startswith("# "):            h1 = line[2:]        else:            sections[-1][1] = (sections[-1][1] + " " + line).strip()    return sections def make_children(sections):    children = []    for parent_id, (path, text) in enumerate(sections):        for sentence in text.split("."):            if sentence.strip():                children.append((parent_id, path + ": " + sentence.strip() + "."))    return children def search(question, children, sections):    q = words(question)    best = max(children, key=lambda child: len(q & words(child[1])))    parent_path, parent_text = sections[best[0]]    return best[1], parent_path + ": " + parent_text sections = split_sections(doc_lines)children = make_children(sections)print("Sections:", len(sections), "| child chunks:", len(children)) question = "How long do refunds take?"child, parent = search(question, children, sections)print("Question:", question)print("Child (searched):", child)print("Parent (sent to the model):", parent) def table_row_chunks(header, rows):    return [" | ".join(h + ": " + v for h, v in zip(header, row)) for row in rows] print("")for line in table_row_chunks(["Plan", "Price", "Users"], [["Basic", "10", "5"], ["Pro", "25", "50"]]):    print(line)
Expected output
Sections: 3 | child chunks: 6Question: How long do refunds take?Child (searched): Returns > Refunds: They take 5 business days.Parent (sent to the model): Returns > Refunds: Refunds go to the original payment method. They take 5 business days. Plan: Basic | Price: 10 | Users: 5Plan: Pro | Price: 25 | Users: 50
The question matched one precise sentence, yet the model would receive the full Refunds section with its heading path. Notice that the heading path appears inside every child, so even a bare sentence stays understandable. The table rows each carry their column names, so a retrieved row still makes sense on its own.

Knowledge Check

Q1. In parent-child chunking, which chunks are embedded and searched?
Q2. Why keep the heading path with each chunk?
Q3. How should a large table be chunked?

Key Points

✓
Retrieval prefers small chunks and generation prefers large ones, so separate the search unit from the reading unit.
✓
Parent-child chunking searches children and returns their parent section.
✓
Hierarchical chunking supports several levels, from document down to sentence.
✓
Structure-aware chunking splits at headings and keeps the heading path with each chunk.
✓
Tables, lists and code need special rules so their meaning survives splitting.
✓
Store links such as parent id, neighbors, references and versions, but add them only when evaluation shows a need.
✓
Expanding context too far wastes prompt space, so measure the benefit.

The Big Picture

Good chunking respects how people wrote the document. By searching small pieces and reading larger ones, and by keeping headings and relationships, you give retrieval precision and give the model context in the same system.
Detect structure → Cut at boundaries → Link children to parents → Search small → Read big
Up Next: Embeddings for RAG. With the data side complete, you move to the representation side: what embeddings are, how meaning becomes numbers, and why similar texts end up close together.