How LLMs Actually Work
An LLM predicts the next token over and over. That one idea explains almost everything it does.
You paste a paragraph into ChatGPT and it continues it flawlessly. You ask the same question twice and get two different answers. You give it a giant document and it “forgets” the start. These look like bugs — they’re actually the same mechanism showing through. By the end of this lesson, one simple idea should explain all three.
It predicts the next token — that’s the engine
A large language model does exactly one thing, billions of times: given the text so far, it predicts the most likely next token (a token is a chunk of text — roughly a word or word-piece). It appends that token, then predicts the next, then the next. Fluent paragraphs are just this loop running fast.
There is no separate “understanding” step bolted on. The apparent intelligence emerges from having learned, from enormous amounts of text, which continuations are likely in context.
Prompt: “The capital of France is”. The model assigns probabilities to possible next tokens: “ Paris” (very high), “ a” (low), “ banana” (almost zero). It picks from the top, appends “ Paris”, and continues. No lookup table — just a learned sense of what token most plausibly comes next.
Tokens are the unit — and the cost
Models don’t see letters or words; they see tokens. “cat” might be one token; “tokenization” might be three. Everything is measured in tokens: how much you can send, how much it can reply, and what you pay. A rough rule of thumb: ~4 characters per token in English, so ~750 words ≈ 1,000 tokens.
The context window is its working memory
The model can only “see” a limited amount of recent text at once — the context window, measured in tokens. Everything in the window influences the next prediction; everything that falls outside it is simply gone. That’s why a very long conversation “forgets” the beginning: it scrolled out of the window. The model has no memory between calls beyond what you put in the window.
Temperature is the randomness dial
At each step the model has a probability distribution over next tokens. Temperature controls how adventurously it samples from that distribution. Low temperature → it almost always picks the most likely token (consistent, repetitive). High temperature → it more often picks less-likely tokens (creative, but riskier). That randomness is why the same prompt can give different answers — it’s a feature you can tune, not a glitch.
Imagine the world’s most well-read autocomplete. Your phone suggests the next word from a tiny history; an LLM does the same thing but having read much of the internet, looking at a much longer history (the context window), with a creativity knob (temperature) for how predictable vs. surprising its guesses are.
Why the same question gives two answers: 1. You ask “Give me a startup idea.” 2. After your prompt, many next tokens are reasonable, so the distribution is spread out. 3. With temperature > 0, the model samples — landing on a different high-ish-probability path each time. 4. Two runs → two different but plausible ideas. Set temperature to 0 and it will tend to repeat the single most likely answer. Same mechanism, different dial setting.
This is the reading. The interactive version — active-recall quiz, a hands-on experiment you run in your own AI, and an earned mastery check — is free in the app.
Start this lesson free →