Context Windows: The Working-Memory Budget
A model has no memory between calls — only what fits in its context window right now. Treat that window as a scarce budget, not an infinite scratchpad.
Back in Module 2 (lesson 5) you learned “context is everything” — the model can only use what’s in front of it. This module goes deep on the consequences, and it starts with a hard limit most people bump into painfully: the context window. A model doesn’t remember your last conversation, or even the top of a long document once it scrolls off — it works only with the chunk of text it’s handed this call, and that chunk has a fixed maximum size. Understanding this one constraint reshapes how you build with AI. Hold the question: if the model has no memory of its own, what is it actually “thinking with” on any given request?
The window is the model’s entire working memory
Recall from lesson 2 that a model generates each token from the text so far. That “text so far” lives in the context window: the block of text (measured in tokens — roughly word-pieces, lesson 4) the model can attend to in a single call. It is the model’s entire working memory — there is no hidden store of your history behind it. Everything the model “knows” about your situation right now — the conversation, the documents, the instructions — must physically fit inside that window, and the window has a hard maximum. Go over it and something must be dropped or the request is rejected. This is why a fresh chat doesn’t remember yesterday, and why a long document can overflow: nothing outside the current window exists to the model.
The window is a budget: it costs money and time
Crucially, the window isn’t just limited — it’s expensive to fill. Two costs scale with how much you put in it. Money: you pay per token, so a request stuffing 50,000 tokens of context costs far more than a tight 2,000-token one. Latency: more context takes longer to process, so a bloated prompt is slower to respond. So the context window behaves like a budget you spend on every call, not a free scratchpad to dump everything into. This reframes a lot of design: the goal isn’t to cram in all possibly-relevant text, it’s to fit the right text — which is exactly why retrieval (Modules 4 and 11) exists, to fetch just the relevant pieces instead of pasting the whole library.
Two ways to answer a question over a 500-page manual: • Naive: paste all 500 pages into the window every call → overflows the limit, or costs a fortune and crawls. • Retrieval: fetch the 3 relevant paragraphs and put those in the window → fits easily, cheap, fast, and often more accurate. • Same question — but respecting the window as a budget changes everything.
More context isn’t always better: lost in the middle
Here’s the counterintuitive finish. Even when text fits, piling in more isn’t automatically better — and can be worse. Models tend to attend best to information at the beginning and end of a long context and can lose track of things buried in the middle (often called “lost in the middle”). So a giant context stuffed with marginally-relevant text can dilute the signal, burying the one crucial sentence where the model half-ignores it. The practical craft: put the most important material where the model looks hardest (near the top or bottom), keep context lean and relevant, and resist the urge to “just add more.” This is the through-line of the whole module — managing a finite window well is its own skill, and it’s why the next lessons tackle memory, summarization, and state. A focused window beats a bloated one.
Think of the context window as a desk, not a filing cabinet. The model can only work with what’s laid out on the desk right now — anything filed away in another room (past chats, the rest of the book) simply isn’t there. The desk has a fixed size: pile on too many papers and they fall off the edges. And even within the desk, papers dead-center under a stack get overlooked, while the ones right in front of you and off to the side get read. A good worker doesn’t bury their desk under every document they own — they lay out just the few pages they need for this task, in the spots they’ll actually look. Building with AI is learning to set that desk well, every single call.
Designing within the budget: 1. A support bot needs the user’s question, the relevant policy, and its instructions — a few thousand tokens. Lean and fast. 2. A developer instead pastes the entire policy manual + full chat history every call → slow, pricey, and the key policy gets “lost in the middle.” 3. Fix: retrieve only the relevant policy section, keep a short running summary of the chat (next lessons), and put the instructions and question at the edges where the model attends. 4. Same model, same window size — but treating it as a budget makes it cheaper, faster, and more accurate.
This is the reading. The interactive version — active-recall quiz, a hands-on experiment you run in your own AI, and an earned mastery check — is free in the app.
Start this lesson free →