The Knowledge Problem

A model knows its training data and your prompt — nothing else. RAG is how you give it the rest.

Building with AI · Lesson 12 · 9 min read

You ask an AI about your company’s refund policy and it confidently invents one. It’s not lying — it never had your policy, so it filled the gap with a plausible guess. Two hard limits cause this: a training cutoff and no access to your private data. And yet — attach the policy PDF to the same chat, and the answer is suddenly right. That attach button is the fix in miniature. This module opens it up: how it works, why it sometimes misses, and how to build it yourself. It’s called RAG.

Two hard limits: cutoff and private data

A model’s knowledge is frozen at its training cutoff — it doesn’t know events after it, and it never knew your private data (your docs, your database, today’s numbers). When asked anyway, it predicts a plausible answer — which is how confident hallucinations happen. The model isn’t broken; it’s answering from patterns because it has nothing else to go on.

Put the knowledge in the context

You already know the fix in miniature (Module 2: context is everything). Retrieval-Augmented Generation (RAG) automates it: when a question comes in, fetch the relevant documents and place them in the prompt, so the model answers from your real, current data instead of guessing. The model stays the same; you’re feeding it the right context at the right moment.

And here’s the thing — you’ve already used RAG. When you attach a file to Claude or ChatGPT, or ask questions inside a “project” with saved docs, the tool is quietly fetching relevant pieces of your file into the prompt. The rest of this module opens that black box — so when an answer misses, you’ll know exactly which part to fix.

Worked example

Question: “What’s our refund window?” → RAG searches your policy docs, finds “30 days,” puts that snippet in the prompt, and the model answers “30 days” grounded in your text — instead of inventing “14 days.”

An everyday analogy

A brilliant new hire who has read the whole internet but not a single one of your company’s documents. Ask them about your refund policy and they’ll guess based on “companies in general.” Hand them the actual policy doc first, and they answer correctly. RAG is the assistant who fetches the right doc before answering.

Worked example

Why grounding beats guessing:
1. Bare model: “Our refund window is 14 days” (plausible, wrong).
2. RAG: retrieve the policy snippet “refunds within 30 days,” add to prompt.
3. Model now answers “30 days,” and you can even show the source.
4. Same model, real answer — because it had the fact in front of it.

This is the reading. The interactive version — active-recall quiz, a hands-on experiment you run in your own AI, and an earned mastery check — is free in the app.

Start this lesson free →

← LLM as a Judge Embeddings & Meaning →