The previous essay considered probes and the Logit Lens: ways of reading clues inside a model. This essay follows those clues further, first trying to separate entangled representations, then intervening to test how they contribute to computation.
A sparse autoencoder supplies a candidate vocabulary; activation patching supplies an experimental operation. They can work together, without being scalpels that guarantee the truth as soon as they are picked up.
1. An unsettling analogy↗
Suppose an alien civilisation captures your brain. It can record every neuron's activity and trace signals as they travel, yet it has never experienced what humans call sadness.
An old song plays. The visitors can measure, model, and predict a sequence of changes. How do those records connect to sadness? How does the concept refer to an experience, rather than a list of numbers that vary together? Technical access has not automatically answered the question.
Now reverse the positions. Researchers face a non-human information-processing system and try to describe it in human words. The analogy neither proves that the visitors could never understand sadness nor assumes that the model has a comparable experience. It points to a different difficulty: observing more closely and choosing the right explanatory units are not the same achievement.
Even before experience enters the picture, “What does this neuron represent?” is already difficult enough.
2. Polysemanticity: one neuron, one concept?↗
If every neuron corresponded to one clear concept, the job would be easier: find the Eiffel Tower, find the subject, and draw their connections. One unit, one meaning—the intuition behind monosemanticity.
In practice, a neuron may activate in semantically different contexts. A few highly activating examples do not establish a dedicated location for a concept. The neuron basis may not be the best unit of analysis.
One proposed explanation is superposition: a model represents features along more directions than the activation space has dimensions, rather than reserving one coordinate per feature. Imagine 100 dimensions and more attributes to represent. There can be at most 100 mutually orthogonal nonzero directions. Allowing non-orthogonal directions creates more possibilities, and introduces interference.
More available directions do not mean arbitrarily many concepts can be recovered without loss. Co-occurrence, sparsity, and how decoding handles interference all matter. Polysemanticity is an observation; superposition is one proposed mechanism behind it, not a synonym.
The question shifts slightly: not only “What is this neuron?” but “Which directions offer a meaningful, testable decomposition?”
3. Sparse autoencoders: untangling the thread↗
3.1 More dimensions, fewer active features↗
Anthropic's 2023 work decomposed a 512-neuron layer in a small language model into more than 4,000 features, finding patterns associated with DNA sequences, legal language, Hebrew text, and other contexts. The important change was not a longer list of labels for the original neurons, but a different set of analytical units.
A sparse autoencoder, or SAE, can be imagined as a decompressor. It expands an activation into a larger collection of candidate features, using only a few at a time to reconstruct the input. The vector is not being “compressed into a higher dimension”: there may be more latent variables, while the constraint concerns how many activate or how strongly.
One common ReLU SAE takes the form:
Here , , and usually . The matrices are and , with biases and . One training objective over sampled activations is:
The coefficient trades reconstruction against sparsity. Here the decoder columns are constrained to unit norm, preventing a larger dictionary scale and smaller coefficients from reducing the sparsity penalty by themselves. This is an illustrative baseline, not the only SAE architecture.
“Sparse Autoencoders Find Highly Interpretable Features in Language Models” (ICLR 2024) reports that learned features are, on average, more interpretable than several baselines under the models and automated measures tested. Yet successful reconstruction does not identify a unique, complete set of “true features.”
Gao and colleagues (2024) scaled to an SAE with 16 million latents on GPT-4 activations. Their method uses TopK and other techniques to control sparsity; the baseline objective above should not be mistaken for that experiment's training recipe. A latent count is not a count of verified human concepts. Names and examples begin an interpretation; they do not establish that the model has a feeling.
3.2 Has one tangle become another?↗
Before treating the SAE as a master key, consider a less convenient result.
Chanin and colleagues' 2024 preprint studies first-letter information. A latent appears to represent words beginning with a letter, yet fails to activate on some appropriate words; more specific latents absorb the relevant direction instead. This is feature absorption. It is not simply a story of frequent patterns swallowing rare ones, nor something solved merely by enlarging the dictionary.
A separate work, “Resurrecting the Salmon” (2025 preprint), compares domain-specific and broad-domain SAEs using medical question-answering material and Gemma-2 activations, reporting reconstruction and interpretability benefits in that setting. It supports checking whether the training distribution suits the question. It does not show that every general-purpose SAE is misguided.
Gemma Scope (2024) releases SAEs across multiple layers and sublayers of Gemma 2, making such comparisons easier. The scale of a resource does not establish that every important concept has been separated, or that an unrecovered concept is inherently inexplicable.
The thread metaphor remains useful. Each new strand still raises questions: where does it activate, what does it miss, and how does it affect the computation downstream?
4. Activation patching: from decomposition to experiment↗
4.1 Transplanting an internal state↗
Imagine a detective replacing a frame in one recording with a frame from another, then checking whether the subsequent judgment changes. The image is memorable. But a frame that changes a judgment is not thereby the unique cause of a crime.
Activation patching specifies the operation more precisely: run two inputs, take an activation from one run, place it at the corresponding site in the other, and recompute what follows.
- Choose inputs with a clear behavioural contrast and fix an output metric.
- Run the reference input and cache the activation at the chosen layer, position, or component.
- Run the baseline input, substitute the reference activation at that site, and continue computation.
- Compare reference, baseline, and patched outputs, checking the replacement scope and input controls.
For example, contrasting “The Eiffel Tower is in…” with “The Colosseum is in…” helps trace how the answer depends on subject information. It is not a matter of inserting a false fact and calling any resulting change a discovery of the “country neuron.”
For a metric where higher values mean behaviour closer to the reference, denote the three scores by , , and . One can report:
The normalised recovery is defined only for a nonzero denominator. It is not a probability: values below zero or above one are possible, and a near-zero denominator makes it unstable. Read it alongside the raw scores, intervention site, and input distribution.
A changed output establishes a causal effect of that replacement on the metric. It does not yet show that the component exclusively handles a semantic concept, or that it is the only route.
4.2 How the scalpel changes the conclusion↗
Zhang and Nanda (ICLR 2024) systematically compared corruption methods, metrics, and patching scope, finding that these choices can yield different localisation results. An account must specify what was replaced and what remained, rather than just display its most striking heatmap.
| Operation | What changes | What needs attention |
|---|---|---|
| Ablation | Set to zero, a mean, or another baseline | The baseline may create abnormal states |
| Activation patching | Substitute another run's activation | Input pairs, sites, and scope define the question |
| Path patching | Restrict the information path examined | State which nodes are recomputed or held fixed |
These operations address related questions, rather than forming a simple-to-advanced ranking. Replacing a whole hidden state and patching a specific path are not tests of exactly the same claim.
5. Circuits: connecting the clues↗
Conmy and colleagues' ACDC (NeurIPS 2023) automates part of circuit finding: given data, a behavioural metric, and a computational graph, search for connections that preserve the behaviour. Automating the search does not decide what should be explained, or guarantee a unique, complete mechanism.
Indirect object identification, or IOI, provides a concrete example from the paper's English task:
When Mary and John went to the store, John gave a drink to ___
The completion is Mary. Wang and colleagues' 2022 preprint analyses a GPT-2 small circuit with 26 attention heads in seven classes on this template-based task. Name Mover Heads write name information towards the output; S-Inhibition Heads affect their queries, reducing attention to the subject; Negative Name Mover Heads write against the names they attend to.
The labels summarise observed component behaviour, rather than identifying a literal “subject manager.” The paper also finds compensating backup heads after the main Name Mover Heads are ablated. A route sufficient to preserve behaviour need not include every route available under other interventions.
The ICLR 2024 SAE work discussed earlier also performs finer-grained feature interventions on IOI. Finer units do not automatically mean universally greater accuracy, or justify an hours-to-seconds speed claim without a matched comparison.
When a circuit has been explained, how far does the understanding reach? Is it a mechanistic account that predicts new interventions, or a description tied to the current templates and metric? That question does not dismiss local understanding. It identifies where the next test belongs.
6. The temptation of control↗
Some representations can now be decomposed, some information flows traced, and some behavioural pathways tested. The next question comes naturally: can they be controlled?
Does finding a feature called “lying” let deception be switched off? Does a direction associated with harmful content guarantee permanent distance from harm? Might adjusting a latent associated with creative output also change other abilities?
These are questions, not promises the tools have already fulfilled. The next essay considers representation engineering and knowledge editing: what changes, how long does it hold, and where do the costs fall?
As understanding becomes intervention, the opening alien civilisation returns. The observer not only names the system, but starts changing it according to those categories. One more question follows: who decides what is worth changing, and what would show that the change actually achieved its purpose?