Chunking Strategies in Depth
Lesson 16 said “split on paragraphs, add overlap.” Now meet the real strategies — and how the shape of your documents decides which one to use.
Back in lesson 16 you learned the instinct: the chunk — a slice of a document — is retrieval’s unit of truth, so cut on paragraphs and add a little overlap. That’s the right reflex, but real documents fight back: a legal contract, a code file, a chat log, and a textbook each break in completely different places, and a one-size chunker mangles most of them. This module opens where lesson 16 stopped — with the actual named strategies professionals choose between, and why the document’s shape picks the winner. Hold the question: if “split on paragraphs” is only a default, what do you do when a document has no clean paragraphs — or when its meaning ignores them?
The baseline: fixed-size chunking (and why it’s crude)
Fixed-size chunking is the simplest strategy: cut the text every N characters (or tokens), usually with some overlap so a sentence straddling a boundary survives in both pieces. It’s fast, predictable, and works acceptably on uniform prose. But it’s blind to meaning: it will happily slice through the middle of a sentence, a table, or a function, producing a chunk that’s half one idea and half another — exactly the “averaged-blur embedding” failure from lesson 16 (recall embeddings from lesson 13 turn text into a point in meaning-space; a chunk covering two topics lands between them and matches neither). Fixed-size is the floor you improve on, not the goal.
Respecting structure: recursive and structure-aware chunking
Better strategies use the document’s own structure as the cut lines. Recursive chunking tries a hierarchy of separators in order: first split on the biggest natural boundary (sections), and if a piece is still too big, split it on the next (paragraphs), then sentences, and only fall back to raw characters as a last resort. This keeps chunks as whole as possible while staying under a size limit. Structure-aware chunking goes further by using a format’s known boundaries: split code on function or class boundaries, Markdown on headings, HTML on sections, a transcript on speaker turns. The principle is the same as decomposition from lesson 40 — break things at the seams the content already has, not at arbitrary points.
Chunking a code file three ways: • Fixed-size (500 chars): cuts a function in half — one chunk has the signature, another has the return, neither is understandable alone. • Recursive: tries file → class → function → line; lands each chunk on a whole function. Much better. • Structure-aware: parses the code and puts each function (with its signature and docstring) in its own chunk. Best — every chunk is a self-contained, retrievable unit.
Semantic chunking, and choosing by document shape
Semantic chunking is the most advanced: instead of trusting punctuation or structure, it measures meaning — embedding sentences and starting a new chunk whenever the topic shifts (when consecutive sentences stop being similar). It groups by idea, not by layout, which is powerful for rambling text with no clean structure. The cost: it’s slower and more complex (you embed while chunking). So there’s no universal best — you choose by the document’s shape: fixed-size for uniform prose you just need roughly split; recursive as a strong general default; structure-aware when the format has real boundaries (code, Markdown, transcripts); semantic when meaning wanders across the layout. The mastery move isn’t memorizing one chunker — it’s matching the chunker to the material, then measuring (a later lesson) whether retrieval actually improved.
Think of cutting a long film into clips. Fixed-size chunking is chopping every 5 minutes on the dot — you’ll slice right through the middle of scenes. Recursive chunking is “cut at act breaks; if an act’s still too long, cut at scene breaks; then at shot changes” — you keep whole story beats together. Structure-aware is using the script’s own scene markers to cut exactly at each scene. Semantic chunking is watching the film and cutting wherever the subject changes, even if there’s no obvious marker. Same movie, four very different sets of clips — and which is best depends entirely on how the film was made.
Picking a strategy per document type: 1. A tidy FAQ of short Q&A pairs → structure-aware: one chunk per Q&A. Trivial and perfect. 2. A 200-page PDF report with headings → recursive (section → paragraph): whole ideas, bounded size. 3. A messy meeting transcript with no headings, topics drifting → semantic: cut where the conversation changes subject. 4. A pile of uniform product blurbs, all similar length → fixed-size is fine; fancier chunking adds cost for little gain. The document’s shape, not fashion, chooses the tool.
This is the reading. The interactive version — active-recall quiz, a hands-on experiment you run in your own AI, and an earned mastery check — is free in the app.
Start this lesson free →