Why Evaluation Is the Real Skill

“Looks right” and “is right” are different. Only evaluation tells them apart — at scale.

Building with AI · Lesson 9 · 9 min read

Your AI feature demos beautifully. You ship it. A week later users report it’s wrong 1 in 5 times — on cases you never tried. You were grading on vibes. The single skill that separates people who play with AI from people who build with it is evaluation: knowing, with evidence, whether it actually works. This is where you learn it.

“Looks right” is a trap

A fluent answer feels correct — that’s exactly why LLMs are dangerous to eyeball. Reading a single polished output tells you almost nothing about the other thousand inputs you didn’t try. Evaluation replaces “it looked good” with “it passed 47 of 50 cases, and here are the 3 it failed.” One is a feeling; the other is evidence.

Evaluate against examples, not vibes

The core move: collect a set of representative inputs with known-good expectations (a golden set), run your system against all of them, and measure how many pass an explicit criterion. Now a prompt change is a measurable experiment — did the pass rate go up or down? — instead of a guess. This is the same learning-vs-performance idea from the rest of this platform: a good demo is performance; a passing eval is evidence of real capability.

Tuning a classifier: you have 50 labeled examples. Prompt A passes 41/50; prompt B passes 46/50. Now “B is better” is a fact, not a hunch — and you can see exactly which 4 cases still fail and why.

Catching a silent regression:
1. You “improve” a summarizer’s prompt; the demo looks great.
2. You run it against your 30-example golden set.
3. Pass rate dropped from 27/30 to 22/30 — the change broke a category you didn’t demo.
4. Without the eval you’d have shipped a regression that “looked right.” With it, you caught it in minutes.