Build Your First Eval

An eval is just golden examples plus an explicit pass/fail rule. Start tiny; it pays off immediately.

Building with AI · Lesson 10 · 11 min read

You’re convinced evaluation matters — but “build an eval” sounds like heavy infrastructure you don’t have time for. It isn’t. A useful eval can be ten examples and one rule, written in an afternoon, and it will catch regressions for the life of the project. Let’s build the smallest one that works.

Two ingredients: examples + a criterion

An eval needs only (1) a set of golden examples — representative inputs paired with what a good output looks like — and (2) an explicit pass/fail criterion for comparing the actual output to the expectation. The criterion can be exact match, “contains these facts,” valid JSON to a schema, or a rubric. Run your system over the examples, apply the criterion, count passes. That’s an eval.

One golden row for a date-extractor:
input: “Let’s meet next Friday” · expected: a date in YYYY-MM-DD · criterion: output matches the date regex and equals the correct Friday. Pass or fail is now objective.

Start small, grow from failures

Don’t boil the ocean. Begin with 10–20 examples covering the obvious cases plus a few nasty edge cases. Every time a real bug slips through, add it as a new golden example — your eval gets stronger exactly where you’ve been burned. Over time it becomes a regression net that lets you change prompts and models with confidence.

Building a tiny eval for a “refund eligible?” classifier:
1. Write 12 example tickets, each labeled eligible / not-eligible.
2. Criterion: model’s label must equal the gold label.
3. Run → 10/12 pass; inspect the 2 misses.
4. One miss reveals a real prompt gap; fix it, re-run, now 12/12 — and the eval guards it forever.