Short-Term vs Long-Term Memory
A model forgets everything the instant a call ends. Every “memory” you’ve seen it have is an illusion you build by feeding the right text back in.
Last lesson dropped a bombshell: the model’s entire working memory is the context window, and it keeps nothing between calls. But that raises an obvious puzzle — chatbots clearly do “remember” what you said three messages ago, and some apps remember your name across sessions. If the model is truly forgetful, where does that memory come from? The answer reveals that “AI memory” isn’t a feature of the model at all — it’s something you engineer around it, and there are two very different kinds. Hold the question: if the model forgets everything the instant a call ends, how does a chatbot seem to remember your whole conversation?
The model is stateless — memory is an illusion you build
The foundational fact: a model is stateless. Each call is independent — it starts with a blank slate, uses only the text in the window (lesson 48), produces an answer, and then remembers nothing. So how does a chatbot “remember” your earlier messages? Simple and a little surprising: the app replays the whole conversation back into the window every call. When you send your fifth message, the app actually sends the model messages 1–5 all over again, so it looks like continuous memory — but really the model is re-reading the entire chat fresh each time. The “memory” lives in your application, not in the model. This single realization is the key to the whole module.
Short-term memory: the conversation in the window
Short-term memory is exactly that replay: keeping the recent conversation inside the context window so the model has it on hand. It’s simple and immediate — but it inherits the window’s limits (lesson 48). A conversation that grows long enough will overflow the window or get expensive/slow, at which point you can’t just keep everything. Short-term memory is like holding a thought in your head right now: fast and effortless while it fits, but finite — you can only keep so much “in mind” at once, and older things fall away. That looming overflow is the exact problem the next lesson (summarization) solves.
A 5-message chat, under the hood: • You send message 5. The app doesn’t send just “message 5” — it sends the whole transcript (1,2,3,4,5) into the window. • The model reads all five fresh, answers, then forgets everything. • Message 6? The app sends 1–6. The “memory” is the app re-supplying history — short-term memory is just the conversation riding along in the window each call.
Long-term memory: an external store you retrieve from
What about remembering your name next week, or facts from a chat months ago — things far too big and old to keep in the window? That needs long-term memory: an external store (a database or vector store, Module 4) where the app saves important facts, and later retrieves the relevant ones to place back into the window when needed. Notice the mechanism — it’s exactly retrieval (Modules 4 and 11): the app searches its memory store for what’s relevant to this moment and injects just those pieces, rather than replaying everything. So the two memories differ cleanly: short-term = the recent conversation kept live in the window (limited, immediate); long-term = a durable outside store you selectively fetch from (vast, persistent). Both ultimately funnel through the same window — because, as always, the model can only ever use what’s in front of it right now. Memory in AI is not something the model has; it’s an engineering pattern you build around a forgetful model.
Imagine a brilliant consultant with total amnesia: the moment you leave the room, they forget you ever met. To work with them, you do two things. Each meeting, you hand them a transcript of everything said so far today so they’re caught up — that’s short-term memory, and it works until the transcript gets too thick to read in one sitting. And you keep a file on them with the important long-term facts — your name, past decisions — pulling out the relevant pages to hand over whenever they’re needed again; that’s long-term memory. The consultant never actually remembers anything. You remember, and re-brief them every time — which is precisely how “AI memory” works.
Two kinds of memory in one assistant: 1. Within a chat: the app replays the recent messages into the window each call → short-term memory, so it “follows” the conversation. 2. Across sessions: when you say “I prefer metric units,” the app saves that to a store; next week it retrieves and injects “user prefers metric” → long-term memory. 3. Both end up as text in the window at call time — the model itself still remembers nothing. 4. Design choice: recent + immediate → keep in window (short-term); important + durable → save and retrieve (long-term).
This is the reading. The interactive version — active-recall quiz, a hands-on experiment you run in your own AI, and an earned mastery check — is free in the app.
Start this lesson free →