Calibration: Being Right About Being Unsure
A great forecaster isn’t the most confident one — it’s the one whose “70% sure” turns out true about 70% of the time.
Two pundits make predictions all year. One is loud and certain and right about half the time; the other hedges with percentages and, when you check, is right exactly as often as they said they’d be. Which is the better forecaster? Our culture rewards the confident one — but there’s a precise, measurable sense in which the hedger is far more valuable, and it’s the quality that separates real forecasters from talking heads. Hold the question: what makes a probability forecast “good,” beyond just being right a lot?
Calibration: confidence should match reality
A forecaster is well-calibrated when their confidence matches how often they’re right: of all the things they call 70% likely, about 70% should actually happen; of their 90%-calls, about 90%. That’s it — calibration is the honesty of your confidence. A calibrated “70%” is a promise the world keeps. Notice this is separate from being right a lot: a weather app that says “70% rain” and it rains 7 of those 10 days is perfectly calibrated even though it was “wrong” 3 times. The uncertainty was the point, and it was honest.
Overconfidence is the default bug
Almost everyone is overconfident: the things people call “90% sure” happen far less than 90% of the time; “I’m certain” is wrong distressingly often. Confidence feels like knowledge but usually isn’t (recall the metacognition thread — “sure and wrong” is the most useful moment to learn from). Loud certainty is cheap and persuasive, which is why pundits trade in it. The antidote is to treat your confidence as a quantity you can be wrong about — and to notice that saying “70%” when you mean it is braver and more useful than saying “definitely.”
Two forecasters over 100 predictions: • Loud Larry says “definitely” on all 100; 62 come true. He was “confident” but his confidence was a lie — 100% claimed, 62% delivered. Badly overconfident. • Calibrated Cara spreads her calls: her “60%” predictions hit ~60%, her “90%” hit ~90%. She’s “wrong” plenty, but every number she gives is trustworthy. Cara is the real forecaster.
You can measure it — and get better
The beautiful part: calibration is checkable. Write down predictions with probabilities, wait, and score them — group all your “70%” calls and see if ~70% came true. That feedback loop (a “track record”) is how forecasters improve: most discover they’re overconfident and learn to widen their uncertainty. This is why the honest forecasters in this track keep giving ranges and probabilities instead of bold single calls — not because they’re wishy-washy, but because a calibrated “I’m 65% on this” carries real, testable information, while “trust me, it’s certain” carries almost none. Being right about how unsure you are is a skill you can train.
Think of a basketball player calling their shots. A braggart yells “nothing but net!” every time and makes half — his words are worthless because they never match reality. A calibrated player says “I make about 40% from here” and, sure enough, sinks about 40% — so when they say “this one’s 80%,” you believe them and can bet on it. The calibrated player misses plenty, but every number out of their mouth is money. Good forecasting is being that player: your stated confidence is only worth something if the world keeps the promise it makes.
Grading your own calibration: 1. Over a month, log 20 predictions, each with a probability (“65% this launches on time,” “80% this deal closes”). 2. When outcomes land, bucket them: take all your ~70% calls together. 3. If about 70% came true, you’re calibrated at that level; if only 45% did, you were overconfident and should widen your uncertainty next time. 4. Repeat, and your numbers become trustworthy — to others and to yourself. You didn’t get more certain; you got more honest, which is what actually helps you decide.
This is the reading. The interactive version — active-recall quiz, a hands-on experiment you run in your own AI, and an earned mastery check — is free in the app.
Start this lesson free →