In the previous essay, we entered the Transformer’s “library without a librarian” and looked at how it connects one page to another. Now for the deeper questions: how large should this library be? Is it really “thinking”? And what, exactly, does it have in common with our brains?
Chapter 1: The pull of scale↗
1.1 Following a power law↗
How large should a model be? It sounds like an engineering question, yet it can produce a peculiar philosophical vertigo. If a little more computation can take us another step forward, what exactly are we approaching?
In Scaling Laws for Neural Language Models, Kaplan and colleagues studied how test loss changes with model size, data, and compute. Under particular conditions, including sufficient data and training, their parameter-scaling fit can be written in simplified form as:
Here, is the number of non-embedding parameters, is a fitted constant, and is test-set cross-entropy loss. Loss measures how well the model predicts subsequent tokens; it is not simply the proportion of questions it gets wrong.
The exponent is small. Substituting the condition “halve the loss” into the same fitted relationship gives:
That is roughly nine thousand times as many parameters, if we follow this curve algebraically.
A curse, and a blessing.
Each step can become more expensive. Yet within the range measured, the curve is still descending.
A curve that keeps descending can look like a road to endless progress. What it gives us first, though, is a regularity under particular experimental conditions. Further along the road, we need to measure again.
1.2 Chinchilla: perhaps the model has not read enough↗
In 2022, Hoffmann and colleagues’ Chinchilla study sharpened the question: with a fixed training-compute budget, how should we divide resources between parameters and data?
Their results suggested that some large models of the time had been trained on too little data for their size. Within the regimes they studied, larger compute budgets were best spent increasing both parameters and training data.
| Model | Parameters | Training tokens |
|---|---|---|
| Gopher | 280 billion | 300 billion |
| Chinchilla | 70 billion | 1.4 trillion |
Using roughly the same training compute as Gopher, Chinchilla combined a smaller model with more data and performed better on many of the paper’s evaluations.
Chinchilla’s run used about 20 training tokens per parameter. That number belongs to the study and its configuration, rather than serving as a universal recipe. The more useful lesson is that parameters and data have to be considered within the same budget.
1.3 Cheap to train, cheap to use↗
A trained model may go on to answer a very large number of requests. The configuration that makes training most economical may therefore differ from the one that minimizes costs over its working life.
In its 2024 Llama 3 release, Meta reported that its 8B and 70B models were still improving after training on around 15 trillion tokens. It also described the trade-off: a larger model might reach a given performance with less training compute, while a smaller model could justify additional training through lower inference costs.
For a model that will be used repeatedly, spending more compute up front to improve a smaller model can make economic sense. Whether it does depends on the number of future requests, their tasks, and the hardware and latency requirements.
“How large should the model be?” now becomes a more concrete question. Where do we want to spend the computation, and when do we want that investment to pay off?
Chapter 2: When does an ability appear?↗
2.1 Emergence is also a measurement question↗
When scale increases smoothly but an ability seems to cross a threshold, we call it “emergent”. Wei and colleagues helped frame this as a research problem. Schaeffer and colleagues then showed how nonlinear or discrete scoring can itself create the appearance of a sudden change.
Imagine a question that earns a point only when the entire answer is correct. A model might be getting progressively closer while its score remains at zero, until it finally gets everything right. The step in the score need not mark an equally abrupt change in the underlying computation.
This does not establish that every change in ability is an artifact. Changes in capability, changes in measurement, and explanations of internal mechanisms are separate questions. Continuous measures, or evaluations of the smaller steps within a task, can help us work out which layer is changing.
I explore this distinction further in “Emergent Abilities: What Are We Actually Measuring?”.
2.2 Saving some computation for the answer↗
Alongside larger models and more training data, we can spend additional compute on a particular problem: generate several candidate answers, check intermediate steps, or revise an earlier attempt using feedback.
Snell and colleagues’ 2024 study investigated how to allocate this test-time compute. In compute-matched comparisons, suitable strategies could outperform a model with roughly 14 times as many parameters on problems where the smaller model already had a nontrivial chance of success.
The conditions matter: problem difficulty, the base model’s capabilities, and the search or verification strategy all affect the result. It cannot be reduced to “thinking longer always makes up for being smaller”.
It reminds me of the line from the Analects about the need to combine learning with reflection. Applied to machines, that is an analogy. The technical question is more concrete: for this problem, should the extra compute go into training a stronger model, or into search, verification, and revision with the model we have?
Chapter 3: From outputs to mechanisms↗
3.1 Where it looks, and what it uses↗
If an answer really is better, what happened inside the model? Mechanistic interpretability tries to turn that question into a testable account of computation.
An attention map is one tempting place to begin. A word receives a large attention weight, and it seems natural to say that it “caused” the answer. But Jain and Wallace demonstrated limits to this interpretation: different attention distributions can produce similar predictions.
The study’s main experiments used BiLSTM and other encoders; it does not directly establish that every Transformer attention head lacks explanatory value. A heatmap can tell us where to investigate. Establishing whether an internal quantity contributes to a particular behavior requires targeted interventions and controls. Reading information out of a representation and showing that the model relies on it are different questions.
I follow this thread further in “Hume’s Ghost: From Readable Information to Causal Evidence”.
3.2 How far can a simple circuit take us?↗
The work on induction heads offers a concrete example. After “A, B” has appeared in a sequence, certain attention heads can help complete a later “A” with “B”. An apparently abstract form of learning from context now has a computational path we can trace.
The strength of the evidence varies with the setting. The authors presented strong causal evidence in small, attention-only models; in larger models containing MLPs, their evidence was primarily correlational.
What fascinates me is the chance to unpack “it seems to have learned” into smaller questions. Which computations support which behaviors? In what models and on what inputs? How much can this circuit explain, and where does the explanation stop working?
3.3 How do features enter behavior?↗
The toy-model work on superposition offers another approach. With sufficiently sparse features, a network can represent more features in a limited-dimensional space through overlapping directions, at the cost of some interference. A neuron need not correspond to one clean, independent concept.
Sparse autoencoders (SAEs) try to decompose activations into more interpretable features. In its 2024 study, Anthropic extracted millions of features from a middle layer of Claude 3 Sonnet and demonstrated cases in which amplifying a feature changed the output accordingly. Strongly amplifying a “Golden Gate Bridge” feature made the model bring up the bridge in many otherwise unrelated answers.
This gives us a concrete experimental route into how representations participate in behavior. Naming a direction “Golden Gate Bridge” is still the beginning of an explanation. We also need to study how it works with other features, how specific the intervention’s effects are, and whether those effects hold across contexts.
Parts of the black box are becoming experimentally accessible. There are still many corridors we have not mapped.
Chapter 4: The library and the human world↗
4.1 Could different paths lead to a similar structure?↗
Borges imagined a library containing every possible book, its librarians searching the corridors for the one that could explain everything. The Transformer brings that image to mind: shelves become vectors; corridors become connections between information. It remains a metaphor. A language model does not actually store every possible book.
What draws me in is another possibility. In learning to solve prediction problems in language, might an artificial system acquire structures that also matter for human cognition?
Schrimpf and colleagues’ 2021 study compared 43 artificial neural-network language models on their ability to predict human language-processing data. The study included two fMRI datasets and one intracranial electrophysiology (ECoG) dataset. Strong Transformer models approached the estimated noise ceiling on some of these language tasks.
The main score was the correlation between predicted and recorded neural responses, divided by an estimated noise ceiling. It is not a direct percentage of all brain activity explained, nor does it establish that the model’s internal mechanisms or subjective experience match ours.
Imagine two people, one raised in Tokyo and the other in Buenos Aires. They have never met, yet they draw similar patterns. The resemblance invites questions. Did they encounter similar constraints, learn similar relations, or did our comparison pick out only part of what each had drawn?
I like this kind of encounter. Similarity in prediction opens a door to comparison. Once inside, we still have to investigate which structures, if any, correspond across the two systems.
4.2 The two faces of attention↗
“Attention” belongs to both psychology and machine learning. You hear your name across a noisy room, and the other voices seem to recede. The word carries a familiar experience of being in the world.
In a Transformer, scaled dot-product attention is a specified operation: queries and keys produce matching scores, which weight the value vectors.
Here, , , and are queries, keys, and values, and is the key dimension. The equation specifies how information is weighted and passed on. It does not, on its own, tell us what human attention feels like.
Both uses suggest selection and the allocation of information, which makes the analogy productive. A shared name can also carry more meaning than we intended. A model can speak fluently about hunger, childhood, or being loved. We still need to ask what relations it has learned about those experiences, how it uses them, and what we can actually test about experience itself.
Somehow, we understand what the machine says. That is remarkable enough to deserve more precise questions.
Chapter 5: The labyrinth of memory↗
5.1 Where does a model put the past?↗
Back in the library, saying that a Transformer’s memory is “all external” bundles together several different things.
Training changes weights, carrying patterns learned from data into the model. During ordinary inference, fixed weights can make use of information in the current context. External databases or saved conversations can also be retrieved into that context. Further training or specialized update mechanisms may change the parameters again.
We therefore need to distinguish patterns learned in weights, information in the current context, and externally stored records. An assistant’s ability to resume yesterday’s conversation does not establish that the conversation updated its weights. Nor does it mean every memory-related task depends on an external record.
5.2 Remembering as reconstruction↗
Human memory is also more complicated than replaying a recording. In their review of remembering the past and imagining the future, Schacter, Addis, and Buckner discuss how the two draw on much of the same neural machinery, using stored information for reconstruction and simulation.
I find that perspective compelling. Memory may matter not only because it preserves what happened, but because it lets the past participate in the present and helps us imagine what has not happened yet. This does not require the stronger claim that every act of remembering completely rewrites the original trace.
If a machine retrieves past records, organizes them into a narrative, and uses them to alter its answer, we can compare that function with ours. Similar functions still leave open questions about implementation, embodiment and lived history, and subjective experience.
“The hippocampus is like a vector database” and “the self is like a system prompt reloaded every morning” are arresting thoughts. I would keep them as invitations to investigate. Neither sentence gives us an adequate account of the hippocampus or the self.
Chapter 6: The ghost in the Chinese room↗
6.1 From answering to understanding↗
In Minds, Brains, and Programs, John Searle proposed the Chinese room. Someone who does not know Chinese follows rules for manipulating Chinese symbols, yet produces appropriate answers from the perspective of people outside. The thought experiment asks whether executing a program is sufficient for understanding.
Large language models give this question new material. Their parameters are learned through training, and their computations are much more complex than a handwritten rulebook. But “the rules were learned” does not, by itself, settle Searle’s philosophical challenge.
Empirical work can make the questions more specific. The Reversal Curse study found that, in some fine-tuning settings, learning a relation in one direction did not automatically enable a model to answer it in reverse. Learning that a particular person wrote a particular book, for example, did not guarantee that it could name the author when given the book. This concerns retrieving a relation in reverse; it does not assume that every statement “A is B” is logically reversible.
Failures like this reveal limits to generalization. Successful transfer can likewise provide evidence of a particular capability. To interpret either, we need to specify which aspect of “understanding” we are testing.
6.2 What kind of understanding are we trying to explain?↗
When someone uses language appropriately, handles unfamiliar situations, and participates in a conversation, we usually take those behaviors as evidence of understanding. With machines, I do not want the absence of an intuitively recognizable little person inside to end the discussion before it begins.
But language use, internal computation, correspondence with the brain, and subjective experience do not become established together merely because we call them all “understanding”. This is the question map I use to keep them apart:
| Level | How we might investigate it | What this evidence alone does not establish |
|---|---|---|
| Behavioral capability | New tasks, transfer, counterfactuals, and failure cases | The specific internal mechanism |
| Internal mechanism | Features, circuits, interventions, and appropriate controls | The same implementation as a human brain |
| Cognitive correspondence | Predictions of human behavior and neural data across conditions | The same subjective experience |
| Subjective experience | First clarify the concepts and testable criteria | It cannot be inferred directly from fluency or representational similarity |
This table organizes questions; it does not describe four validated stages of understanding. Evidence at different levels can constrain one another, and may lead us towards different conclusions.
Epilogue: trying to understand its understanding↗
Perhaps understanding was never simply an object hidden somewhere, waiting to be found. It also shows itself in how a system uses what it has learned, where it can transfer that learning, and where it repeatedly fails.
That does not make the question easy. It makes me less willing to let a single “yes” or “no” cover up so many details worth investigating.
We entered the library wanting to know whether it understands. A little further inside, we find ourselves asking how we recognize understanding at all—and where we want an explanation to take us.