Multi-Hop Retrieval
Some answers can’t be fetched in one search — you have to retrieve, learn something, and use it to search again. Retrieval becomes a loop.
Every technique so far assumes one retrieval finds the answer: fix the chunks, fix the query, search once, done. But some questions can’t work that way. Ask “Who leads the team that built our billing system?” and there’s no single document with that answer — one doc says which team built billing, another says who leads that team. You need to find the first fact to even know what to search for second. Single-shot RAG is structurally incapable of this, and it fails in a way that looks like a retrieval bug but isn’t. Hold the question: how do you answer a question when finding the answer requires first finding something else?
Why one search isn’t enough
Some questions need chained facts: the answer depends on an intermediate fact you don’t have yet. “Who leads the team that built billing?” hides two hops — (1) which team built billing, then (2) who leads that team. A single retrieval embeds the whole question and looks for one passage that matches, but no such passage exists, so it returns something plausible-but-wrong (often the leader of whatever team the question’s words most resemble). This isn’t fixable by better chunking or query rewriting (lessons 44–46) — those help find a fact that exists in one place. Here the problem is different: the answer is assembled from facts in different places, and you can’t search for the second until you know the first.
Multi-hop: retrieve, learn, retrieve again
Multi-hop retrieval (also called iterative retrieval) turns retrieval into a loop instead of a single shot: retrieve for what you can search now, read the result, use what you learned to form the next query, and repeat until you have every fact the answer needs. This should feel familiar — it’s the agent loop from lesson 36 (perceive → decide → act) applied to search, and each new query is a small act of query rewriting (lesson 46) driven by the last result. The compound question is handled by decomposition (lesson 40): break it into ordered sub-questions, where each hop’s answer unlocks the next hop’s query. Retrieval stops being a lookup and becomes a short investigation.
Answering “Who leads the team that built billing?” in two hops: • Hop 1 — search “which team built the billing system” → retrieve a doc: “Billing was built by the Payments team.” New fact: Payments team. • Hop 2 — now searchable: “who leads the Payments team” → retrieve: “Payments is led by Dana Wu.” • Assemble: the answer is Dana Wu. Neither hop alone could have found it; the first hop’s result made the second hop’s query possible.
The cost, and when to reach for it
Multi-hop is powerful but not free: each hop is another retrieval and another model call to decide the next query, so it’s slower and pricier, and — like any loop (lesson 20) — it needs a stop condition so it doesn’t wander forever chasing tangents. So don’t make everything multi-hop. Use single-shot retrieval for the common case where the answer lives in one place, and reserve multi-hop for genuinely compositional questions — “the X of the Y that did Z,” comparisons across entities, “trace this back to its cause.” The tell is linguistic: if answering the question out loud would make you say “well, first I’d need to find out…”, it’s multi-hop. Matching the retrieval shape to the question’s shape — one fact vs assembled facts — is the mastery skill this module has been building toward.
It’s the difference between looking up a word in a dictionary and doing detective work. A single-shot search is the dictionary: one lookup, answer’s right there. But “who leads the team that built billing?” is a detective case — you interview one person to learn which team was responsible, and only then do you know whose office to visit next. You couldn’t have walked straight to the team lead, because until the first clue, you didn’t even know their name. Multi-hop retrieval is the AI doing that legwork: following each clue to the next, assembling an answer no single document ever held.
Spotting multi-hop vs single-hop questions: 1. “What’s our refund window?” → single-hop; one policy doc has it. Don’t loop. 2. “Which of our two biggest customers signed more recently?” → multi-hop: find the two biggest customers, then look up each signing date, then compare. 3. “What caused the outage that hit the checkout service last March?” → multi-hop: find the March checkout outage, then retrieve its post-mortem cause. 4. Heuristic: if you’d naturally say “first I’d need to find out X, then…”, loop; otherwise one search is cheaper and enough.
This is the reading. The interactive version — active-recall quiz, a hands-on experiment you run in your own AI, and an earned mastery check — is free in the app.
Start this lesson free →