Self-Consistency: Ask, Then Vote

Run the same reasoning prompt several times and take the majority answer — noise cancels, and the reliable answer rises to the top.

Building with AI · Lesson 42 · 9 min read

Chain-of-thought makes a model reason before answering — but it doesn’t reason the same way every time. Ask the identical hard question twice and you can get two different chains, sometimes two different answers, because there’s randomness in how it generates. On a genuinely hard problem, any single run might stumble down a bad path. There’s a simple, powerful fix borrowed straight from crowds and elections. Hold the question: if one run is unreliable, how could running the same prompt several times make you more sure?

One run is a sample, not the truth

A model’s output has built-in randomness (recall from Module 1 that it picks among likely next tokens, not always the single top one). So a chain-of-thought run is really one sample from a cloud of possible reasoning paths. On easy questions nearly all paths land on the same answer, so one run is fine. On hard questions the paths scatter — some reason well, some take a wrong turn — and any single run might be the unlucky one. Trusting one run on a hard problem is trusting one roll of a slightly loaded die.

Self-consistency: sample many, take the majority

Self-consistency is the fix: run the same chain-of-thought prompt several times independently, collect the final answers, and take the majority vote. The good reasoning paths tend to converge on the same correct answer, while the flawed paths fail in scattered, different ways — so the correct answer piles up as the plurality while the mistakes split the rest. It’s the wisdom-of-crowds trick you may have met in the Future track: independent errors cancel, and the signal that many paths agree on rises above the noise. Note it needs an answer you can compare across runs (a number, a label, a choice) — so you can count votes.

A tricky word problem, five independent runs:
• Run 1 → 42 · Run 2 → 42 · Run 3 → 37 · Run 4 → 42 · Run 5 → 51
Three runs reasoned their way to 42; the two wrong runs failed differently (37 and 51). Majority vote → 42. Any single run had a real chance of being 37 or 51; the vote surfaces the answer the sound paths keep reaching.

Reliability you buy with calls

Self-consistency isn’t free — running a prompt five times costs roughly five times as much. That makes it a deliberate trade: spend more compute to buy reliability on the problems that warrant it (a high-stakes calculation, a classification you can’t afford to get wrong), and skip it on easy or low-stakes tasks where one run is plenty. It also pairs naturally with decomposition (lesson 40): vote on the uncertain step of a pipeline, not every step. The mature prompter’s instinct isn’t “always vote” — it’s “spend reliability where a wrong answer actually costs you.”

Deciding when to spend on voting:
1. Task: extract a company’s reported revenue from a messy filing — a number you’ll put in a report.
2. One chain-of-thought run gives $4.2B; but the filing is confusing, and you know single runs scatter.
3. You run it seven times: five say $4.2B, one says $42B (a decimal slip), one says $3.9B.
4. Majority → $4.2B, and the outliers reveal the exact way a single unlucky run could have burned you. High stakes + confusable input = worth the 7× cost; for a throwaway summary you’d have just run it once.