Debugging a Prompt
Most “bad AI” is one of four fixable problems. Naming which one is the whole skill.
Your assistant keeps getting one type of question wrong. You reword the prompt ten times and nothing helps. You’re fixing the wrong layer. The issue might be the prompt — or missing context, or bad retrieval, or a genuine model limit. Until you diagnose which, you’re guessing. This lesson gives you the four-way split.
Four failure categories
Almost every failure is one of four:
- Prompt — the instruction is ambiguous or under-specified (fix the wording/format).
- Context — the model never had the facts it needed (supply them).
- Retrieval — in a RAG system, the right document wasn’t fetched (fix search/chunking).
- Capability — the task is genuinely beyond the model or its training cutoff (use tools, a better model, or a different approach).
Diagnosing the category tells you where to spend effort — and stops you from rewording a prompt that was never the problem.
Symptom: the bot gives outdated prices. Test: paste the current price into the prompt — does it answer correctly? If yes → it was a context/retrieval problem (it lacked the fact), not a prompt-wording one. A single targeted test isolates the layer.
Isolate by testing one variable
Debug like an engineer: change one thing and observe. Hand the model the missing fact directly — if it now succeeds, the issue was context/retrieval, not the prompt. Ask it to reason step by step — if accuracy jumps, it was a reasoning/prompt issue. If nothing helps even with perfect inputs, you’re likely at a capability limit. Each test rules out a category.
It’s medical triage. “The app is broken” is a symptom, not a diagnosis. A good doctor runs one test at a time — is it the prompt (instructions), the context (information), the retrieval (did we even fetch the right chart?), or a true limit? Treating the wrong cause wastes everyone’s time.
A summarizer keeps missing a key clause: 1. Hypothesis A (prompt): reword to “include any payment terms.” No change. 2. Hypothesis B (context): check — was the clause even in the text sent? It was truncated out. 3. Root cause: context (the clause never reached the model), not prompt wording. 4. Fix: stop truncating / chunk better. You found it by testing one variable at a time.
This is the reading. The interactive version — active-recall quiz, a hands-on experiment you run in your own AI, and an earned mastery check — is free in the app.
Start this lesson free →