The thing a language model is trained to do is not merely similar to what a brain is doing. It is the same quantity, measured the same way. Knowing exactly where that stops being true is what makes the rest of the arc possible.
Navigation: ‹ Prediction Is Error Minimization · An LLM Has a Loop But Not a Life ›
A language model is trained on a very simple ritual.
Show it a stretch of text. Hide the next token. Ask it what comes next. Then score it by how much probability it assigned to the answer that actually appeared.
If the model expected the answer, the score is small. If it treated the answer as unlikely, the score is large. If it practically ruled the answer out, the score is enormous. Then the model is adjusted so that, next time it sees a similar pattern, it is a little less surprised.
That is prediction-error minimization.
Not “like” prediction-error minimization. Not “analogous to” prediction-error minimization. Not a cute metaphor borrowed from neuroscience. The quantity being reduced is surprise: the mismatch between what the model expected and what the world of text delivered. In the language-model case, the world is a corpus and the event is the next token. In the biological case, the world is richer and the event may be light, sound, pressure, hunger, pain, temperature, or another creature's motion. But the basic score has the same shape: what did you expect, what happened, and how badly did your expectation fit?
That is why the comparison matters. If next-token prediction were merely a parlour trick, the rest of this movement would be easy. We could dismiss the model and return to living things. But the model is doing something real. It is participating in the same mathematical family of operations that the previous page described in prose.
The honest argument begins only after we grant that.
3. A map is not a beetle
Now, a map can make guesses too.
If you drew the beetle's whole world on a piece of paper — the bark, the stones, the two tunnels — and then you asked the paper what was down the new tunnel, the paper could give you an answer. A good answer, even. Paper with a great many tunnels drawn on it can be right about tunnels remarkably often.
But here is the difference, and it is the whole difference.
Nothing at all happens to a map that guesses wrong.
The beetle was the one who would have to crawl.
A map is not a beetle.