Summarization & Compaction
When a conversation outgrows the window, don’t drop the old parts — compress them into a running summary that keeps the meaning at a fraction of the tokens.
Last lesson left a ticking clock: short-term memory replays the conversation into the window, but a long enough chat overflows the window (lesson 48). So what do you do when the transcript no longer fits — just delete the oldest messages and let the model forget them? That works, but it’s crude: you might drop the exact detail that mattered (“the user’s deadline is Friday,” buried in message 2). There’s a smarter move that keeps the meaning while shedding the tokens. Hold the question: when a conversation gets too long to fit, how do you make room without the model losing the important things that were said?
Compaction: replace old turns with a summary
The key technique is summarization (also called compaction): when the conversation grows too long, take the older portion and replace it with a compact summary of what happened, then keep that summary — plus the recent messages verbatim — in the window. Instead of a 5,000-token history, you now carry a 300-token summary that preserves the gist: decisions made, facts established, the user’s goal. The model reads the summary as if catching up from notes, and the recent turns in full detail. You’ve freed most of the space while keeping most of the meaning — a far better trade than bluntly deleting the old turns. (Fittingly, the summary is usually written by a model — you ask it to summarize the conversation so far.)
A long support chat, compacted: • Turns 1–20 (4,000 tokens): lots of back-and-forth establishing the user’s setup, the error, and two things already tried. • Compact them → a 250-token summary: “User on version 2.3, hitting error X on export; already tried restarting and reinstalling; wants a fix by Friday.” • Window now holds: that summary + the last few verbatim turns → fits easily, and the model still knows everything that matters.
The tradeoff: compression always loses something
Be honest about the cost: summarizing is lossy compression. A summary is smaller precisely because it throws detail away, and you can’t know in advance exactly which discarded detail a future turn will need. Summarize too aggressively and the model “forgets” a specific number, a name, an exact phrasing that later turns out to matter; summarize too little and you barely save space. So compaction is a dial, not a free win — the art is compressing the stale, settled parts hard while keeping recent and likely-relevant details intact. This is the same “keep what matters, drop the noise” judgment as good chunking (lesson 44) and lean context (lesson 48), now applied to a conversation’s history over time.
The hybrid pattern: summary + recent + retrieval
In practice, robust apps combine all the module’s tools into one layered memory. (1) Keep the recent messages verbatim (short-term memory, full detail where it matters most). (2) Keep a running summary of the older conversation (compaction, for the gist). (3) For specific facts too important to risk losing to summarization, save them to a long-term store and retrieve them when relevant (lesson 49). This layering directly reflects the context window being a budget (lesson 48): spend full tokens on what’s fresh, cheap summarized tokens on the settled past, and pull exact facts back from storage only when needed. The takeaway for the module: managing a conversation over time isn’t about a bigger window — it’s about intelligently deciding, every call, what deserves the space.
Think of taking meeting minutes. You don’t keep a word-for-word transcript of a year of meetings on your desk — it wouldn’t fit and you’d never find anything. Instead you keep detailed notes of this week’s meeting, a short summary of the previous ones (“agreed on the budget, picked vendor X, deadline Friday”), and a filing cabinet for the few critical documents you might need to pull exact figures from later. Summarizing the old meetings loses some detail — that’s the price of a desk that fits — but a good summary keeps every decision that still matters. Managing an AI conversation over time is exactly this discipline of minutes, summaries, and files.
Layered memory in a long-running assistant: 1. Recent 6 messages: kept verbatim in the window (full detail). 2. Everything older: compacted into a running summary that updates as the chat grows. 3. Critical facts (“account ID 4471,” “allergic to penicillin”): saved to a long-term store and retrieved when relevant, never trusted to the lossy summary. 4. Result: the conversation can run indefinitely, always fitting the window, while the things that matter survive — recent detail, summarized gist, and exact facts on demand.
This is the reading. The interactive version — active-recall quiz, a hands-on experiment you run in your own AI, and an earned mastery check — is free in the app.
Start this lesson free →