Chain-of-Thought & Tree-of-Thought

Letting a model reason out loud before answering makes it far better at hard problems — and branching several lines of reasoning helps even more.

Building with AI · Lesson 41 · 10 min read

Ask a model a tricky multi-step question and demand just the answer, and it often blurts a confident wrong one. Ask the same model to “think it through step by step” first, and its accuracy on hard problems jumps dramatically — for free. You saw this briefly in Module 2; now we go deeper into why it works and how to push it further when a single line of reasoning isn’t enough. Hold the question: why would letting a model ramble before answering make it more accurate, not less?

Why chain-of-thought works

Recall from Module 1 that a model generates one token at a time, each based on the text so far. When you force it straight to a final answer, it has to compress all the reasoning into that single leap — and complex reasoning doesn’t fit in one leap. Chain-of-thought (CoT) — “think step by step” — lets the model write out intermediate steps, and each written step becomes context the next step can build on. It’s literally giving the model room to work: the reasoning on the page is scratch paper it then reads to reach a better answer. This is decomposition (last lesson) happening inside one response.

Use it deliberately — and know its cost

Reach for CoT on anything with multiple steps: math, logic, planning, careful analysis. Ways to invoke it: simply ask (“reason step by step, then give the answer”), or show a few worked examples that include their reasoning (few-shot CoT). Two honest caveats: it costs more tokens (all that reasoning isn’t free), and — subtly — the written reasoning is a rationalization the model finds helpful, not always a faithful transcript of “how it really thought.” It reliably improves answers; just don’t treat the explanation as a perfect audit trail.

Same question, two prompts:
• “A bat and ball cost $1.10; the bat costs $1 more than the ball. How much is the ball? Answer only.” → the tempting wrong “$0.10.”
• “…Reason step by step, then answer.” → “If the ball is x, the bat is x+1; x + (x+1) = 1.10; 2x = 0.10; x = $0.05.” The scratch work catches the trap.

Tree-of-thought: branch when one path isn’t enough

Some problems have a dead end halfway down a single chain — the model commits to one approach and can’t back out. Tree-of-thought (ToT) generalizes CoT: instead of one line of reasoning, the model (or your code) explores several branches, evaluates which look promising, and pursues the best — backtracking from dead ends, like exploring a maze rather than walking one corridor. It’s more expensive and more complex to orchestrate, so save it for genuinely hard search/planning problems where a single chain gets stuck. The through-line of this module: give the model room to reason (CoT), and when one path isn’t enough, let it explore many (ToT).

Choosing the right tool for a task:
1. “Classify this review’s sentiment.” → simple; no reasoning needed, just answer (CoT would waste tokens).
2. “Solve this multi-step word problem.” → chain-of-thought: reason step by step, then answer.
3. “Plan a route visiting these constraints, where early choices close off options.” → tree-of-thought: explore branches, evaluate, backtrack.
4. Matching the technique to the problem’s difficulty — none, CoT, or ToT — is the skill; more reasoning isn’t always better, it’s better when the problem has steps or dead ends.