Making RAG Good

A working RAG pipeline isn’t a good one — retrieval quality is the ceiling. Here are the levers that raise it, and when to skip RAG entirely.

Building with AI · Lesson 16 · 13 min read

Your RAG bot from last lesson works — on easy questions. But ask for order “#A1043” and it returns generic “how returns work” fluff (the exact code got ignored). Ask something whose answer spans two paragraphs and you get half an answer (each chunk was cut mid-thought). Same pipeline as L15 — so why the misses? Because “it retrieves something” and “it retrieves the right thing” are different problems, and the gap between them is where good RAG is won.

Chunking: how you cut the docs decides what can be found

Retrieval returns chunks, so the chunk is your unit of truth — and how you split documents quietly decides everything. Chunks too big bury the one relevant sentence in noise and produce a fuzzy, averaged embedding that matches nothing well. Chunks too small lose the surrounding context needed to make sense of them. The fix: split on natural boundaries (paragraphs, sections), keep each chunk about one coherent idea, and add a little overlap between chunks so an answer that straddles a boundary isn’t sliced in half.

A refund policy chunked two ways:
• Bad: a 5-page document as ONE chunk → its embedding is a blur of every topic; “refund window” barely registers.
• Bad: split every sentence → “within 30 days” lands in a chunk with no hint it’s about refunds.
• Good: one chunk = the “Refunds” paragraph, with a sentence of overlap into the next — coherent, self-contained, easy to match.

Hybrid search + reranking: two fixes for “close but wrong”

Hybrid search runs two retrievers and merges them. Vector (semantic) search nails meaning but can fumble exact tokens — an order code, a SKU, a person’s name, rare jargon. Keyword search (e.g. BM25) nails those exact matches but misses paraphrases. Run both and combine, and you catch both kinds of question. Reranking adds a second stage: cheaply retrieve a wide net (say the top 20 candidates), then run a slower, more careful reranker that scores each candidate against the query and keeps the best 3–5. Stage one maximizes recall (don’t miss the answer); the reranker maximizes precision (put the best ones on top). Cheap-and-wide, then expensive-and-sharp.

When RAG is the wrong tool

RAG shines for one shape of problem: the answer lives in a body of text you have. It’s the wrong reach when (1) the whole knowledge fits in the context window — just put it in the prompt, no retrieval needed; (2) the answer is live or computed — today’s price, a sum, an account balance — which wants a tool/API call, not a document search; or (3) it’s a reasoning or math task, where there’s no passage to fetch. Knowing when not to add RAG is as valuable as knowing how to build it — bolting retrieval onto a problem it doesn’t fit just adds latency and new failure modes.

Fixing the “#A1043” miss end-to-end:
1. Symptom: vector search returns generic order-status prose; the exact code never matched.
2. Add keyword search (hybrid): “#A1043” exact-matches the specific order record.
3. Merge vector + keyword candidates into a pool of ~20.
4. Rerank the pool against the question; the exact order record rises to the top 3.
5. Augment + generate → the model answers about the right order.
And the straddled-answer question? Re-chunk on paragraph boundaries with slight overlap so the full answer lives in one retrievable chunk. Same L15 pipeline — better inputs, better answers.