Chapter 1: Hume's ghost↗
Are we seeing the truth, or its shadow?
In his 1739 Treatise of Human Nature, David Hume asked where our idea of a “necessary connection” between events comes from.
In Book 1, Part 3, Section 14, he distinguishes observable succession, proximity and repeated conjunction from the expectation produced by custom. Watching sunrise and a rooster's crow recur does not, by itself, reveal the direction or mechanism of their relationship.
This is not a verdict that causation does not exist. It asks what warrants the passage from conjunction already observed to necessity not directly seen.
Nearly three centuries later, the ghost has returned in research on large language models.
A probe reads information from a representation. An attention map gives some positions greater weight. A sparse autoencoder produces a direction with a clear conceptual name.
An unsettling question remains: is this what the model is doing, or a statistical regularity mistaken for a mechanism?
Chapter 2: The forensic examiner's dilemma↗
Imagine a forensic examiner finding a clear fingerprint at a scene. It matches the suspect. The case seems solved.
But wait. The print offers evidence of contact. It does not independently establish when or how contact occurred, much less who committed the crime. Valuable evidence need not answer every question.
The analogous leap in model analysis is that information can be readable without establishing its use in the task. Call this the gap between encoding and use.
Belinkov's 2022 probing review makes the distinction explicit. The probe and the original model are separate systems. After the probe solves a new task on a representation, the original model's dependence on that property remains a question.
2.1 Part of speech: readable where, influential when?↗
Part-of-speech probes at BERT's layers can test how easily nouns, verbs and other labels are predicted from a representation. Saying that BERT uses those labels at that layer goes further.
LEACE, published at NeurIPS 2023, revisits a related experiment. It linearly erases POS information at one layer of bert-base-uncased, continues the forward pass, and measures word-prediction accuracy and loss. Changes before layer eight are mild; the largest effect appears at layer eleven. Linear POS probing performs best earlier.
Readability and behavioral impact under this intervention do not coincide. That does not prove earlier POS information is never used. Later layers may reconstruct it, or other encodings and paths may survive. The paper also compares random projections and other erasure methods: deleting more dimensions can itself damage the task.
The fingerprint is still at the scene. The rest of the evidence cannot be skipped.
2.2 Selectivity: how much did the probe learn?↗
Hewitt and Liang's EMNLP-IJCNLP 2019 paper introduces control tasks to address another confusion: does a good result reflect the representation or the probe's capacity to memorize?
For POS tagging, each word type receives a randomly chosen, fixed label. The same word keeps that label across contexts; labels are not independently resampled for every occurrence. A probe that identifies word types may therefore learn this arbitrary lookup table.
Selectivity compares the two performances:
Here, is linguistic-task accuracy and is control-task accuracy, on the same scale. The difference measures selectivity for this probing setup, not the model's degree of understanding.
Low selectivity warns that high linguistic accuracy can accompany substantial word-identity memorization. It establishes neither the absence of linguistic structure nor whether the original model uses the information.
The paper also finds slightly higher POS accuracy in ELMo's first layer than its second, but greater selectivity in the second. Which layer better “represents” POS? That depends on whether representation means readability, selectivity or a role in the task.
Chapter 3: The lies of attention?↗
If probes raise the problem of readability versus use, attention poses another question: what can a heatmap explain?
Jain and Wallace's NAACL 2019 paper has a blunt title: Attention is not Explanation. Its experiments mainly concern recurrent and related NLP models with attention. Their conclusions cannot be transferred unconditionally to every head in a modern Transformer.
3.1 Weight and importance are different quantities↗
Across the studied tasks, attention weights frequently have weak associations with gradient-based and other importance measures.
Where a model “looks” cannot simply be ranked into a complete account of contributions. Gradients are not an unquestionable ground truth either: they concern local sensitivity, whereas attention weights participate in a weighted computation. Disagreement first requires understanding what each quantity measures.
3.2 Adversarial attention: different allocations, similar answers↗
The authors search for attention distributions far from the original weights that nevertheless produce approximately the same prediction.
This challenges the original heatmap's status as a unique explanation. It does not prove that attention has no causal role. Different combinations of clues can support the same conclusion, and contextual representations can carry redundant information.
Two detectives reaching the same conclusion from different clues need not both be reasoning badly. What becomes less credible is that a list of what they examined fully explains how they solved the case.
3.3 Shuffling attention: what stayed unchanged?↗
Another experiment holds encoded representations fixed, randomly permutes attention weights and measures the output. Many examples show small prediction changes, but some perturbations have substantial effects. Local robustness does not mean all attention can be shuffled freely.
Serrano and Smith's ACL 2019 study likewise finds that larger weights sometimes predict larger effects, without being a fail-safe indicator. Wiegreffe and Pinter, writing that year, argue that the definition of explanation and tests of the complete model matter to the conclusion.
The debate leaves a specific question: under which conditions does this map support a claim about which kind of contribution?
3.4 The deeper problem↗
“Attend,” “decide” and “remember” are convenient human words. They can also slip a familiar psychological picture into an unfamiliar computation.
A model does have a computational process that can be traced. The difficulty is whether human concepts divide it accurately. A component can participate in computation while its color, name and individual weight leave much of its role unexplained.
Hume's reminder returns: when a relationship feels self-evident, which parts are evidence, and which are expectations supplied by the interpreter?
Chapter 4: The dark side of sparse autoencoders↗
Probes have limitations, and heatmaps do not directly explain decisions. Have sparse autoencoders (SAEs), then, solved superposition and found the model's “real features”?
Chanin and colleagues' A is for Absorption, published at NeurIPS 2025, investigates feature absorption.
Imagine a feature apparently corresponding to “starts with S.” On short, however, it does not fire. A more specific short feature does.
The model need not have lost the first-letter information. A more specific SAE latent can include both the short and “starts with S” components in its reconstruction direction, avoiding the cost of activating two latents. The paper explains this sparsity incentive through a hierarchical example and analyzes a corresponding case in Gemma Scope SAEs.
A clean name based on highly activating examples can therefore hide cases that should fall under the interpretation but do not. Naming is easier than checking absence.
Such omissions would matter if comparable features were used to detect dangerous behavior. The first-letter study does not, however, measure a dangerous-behavior detector's failure rate. That risk remains an inference requiring its own tests.
Chapter 5: The Hydra effect, circuits that will not die?↗
McGrath and colleagues' 2023 Hydra Effect study identifies another phenomenon. In its factual-recall setting, ablating an attention layer can increase later layers' contributions, partially compensating for the removal.
Like the mythical Hydra, cutting off one head makes others more important. No retraining or new parameters appear here, though. The response occurs downstream in the same forward computation, and compensation need not be complete.
“Importance” consequently has more than one reading. A component's direct contribution in normal computation can differ considerably from the total change after its removal, when the remaining network can respond.
A small ablation effect therefore does not automatically mean the component was unused. Nor does it prove circuit analysis can never be complete. The object of study is precisely how the original path, compensation and downstream suppression produce the result together.
Chapter 6: The curse of the seed↗
Paulo and Belrose's study, released in 2025, adds uncertainty: SAEs trained on the same data with different initialization seeds can learn partially unmatched latent sets.
Not entirely different sets. The paper finds both shared and unmatched features, with results depending on SAE type, sparsity and the matching rule. A missing one-to-one match also does not establish that one latent lacks an approximate expression through a combination of others.
It nevertheless challenges an attractive assumption: one training run yields the unique, canonical catalogue of concepts.
Which set contains the “real” features? Instability does not settle that question by proving there is no real structure. A more careful question is which structures recur across seeds, and which retain similar interventional roles under a different decomposition.
Chapter 7: Redemption through causal methods?↗
At this point, probes may seem incomplete, attention contested and SAEs unreliable. What can still be trusted?
Observation remains valuable. It proposes candidate explanations. Those explanations must then face intervention.
7.1 From conjunction to change↗
Pearl's causal-inference framework distinguishes observing a value from setting it, while making structural assumptions explicit. Under suitable assumptions, observational data can identify some intervention effects. Causal knowledge does not always require personally conducting an experiment.
Neural networks offer a particular advantage: researchers can fix the weights, rerun computation and directly replace internal states. Whether changing a vector amounts to changing only one concept is still something to establish.
Probes, heatmaps and the Logit Lens offer clues about information and computation. Activation patching, causal tracing and concept erasure test specified changes. Working together, they do more than a division into methods that show only illusion and methods that guarantee truth.
7.2 Activation patching: where does the scalpel cut?↗
Meng and colleagues' ROME work, published at NeurIPS 2022, uses causal tracing to analyze factual recall. The clean–corrupted–restored procedure is straightforward:
- Run the model normally and save internal activations.
- Perturb the subject's input embeddings to reduce the target answer's probability.
- Restore selected clean states during the corrupted run and measure recovery.
For example, a question about the Space Needle's city can trace the continuation “Seattle.” Let be the target answer's probability in the corrupted run and its probability after restoring states :
measures recovery in this setup. A positive value shows that replacing those states helps restore the output. It establishes neither a unique storage location nor an intervention restricted to one semantic property.
The original paper combines activation tracing with weight editing to support a role for middle-layer MLPs in the studied factual associations. Later, Zhang and Nanda, ICLR 2024, show that corruption methods and evaluation metrics can change localization results. A scalpel is valuable when its cut and the measure of its effect are understood.
7.3 LEACE: the scope of “perfect” erasure↗
LEACE gives unreadability a precise constraint. Under its definition of linear prediction and convex loss, prediction cannot improve on the relevant constant baseline. It minimizes average distortion among affine transformations satisfying that constraint.
This is narrower than uprooting a concept while preserving every other piece of information exactly. Nonlinear recovery, downstream reconstruction, distribution shifts and collateral changes to other properties do not disappear with the method's name.
After behavior changes, ask about collateral damage. When behavior does not change, ask about surviving encodings and compensatory paths. Both results add evidence; neither should become a simple yes-or-no verdict.
7.4 Amnesic probing: put behavior back into the question↗
Elazar and colleagues' Amnesic Probing, TACL 2021, asks how original-task predictions change after readable information is removed.
Its BERT analyses find that conventional probe performance does not always correspond to intervention-based task importance. The question moves from how much can be read to what happens after this removal.
The eraser is part of the analysis too. LEACE's re-examination shows why removal scope and collateral damage matter. Not every projection is an ideal operation on a concept.
7.5 Causal abstraction: require an explanation to pass a family of tests↗
The causal-abstraction framework developed by Geiger and colleagues asks whether a high-level model and its alignment with neural representations preserve agreement under corresponding interventions. Matching answers on ordinary inputs is no longer the only commitment.
Wu, Geiger and colleagues' Alpaca study uses Boundless DAS to search for alignments in a rotated representation space. The task is specific: decide whether a price lies within a given interval. Candidate explanations include boundary checks and distance from the midpoint.
At several locations, boundary-check hypotheses achieve interchange-intervention accuracy at or above the model's roughly 85% task accuracy; other candidates score lower. This supports boundary-based computation within the tested scope, rather than a near-perfect account of everything Alpaca does.
An explanation gains strength when it continues to predict on new inputs and interventions. How alignment is learned, how testing is separated, and which alternatives also pass determine how much strength it has.
Chapter 8: The epistemological lesson↗
Return to Hume.
Observation is not understanding, correlation is not causation, and presence is not use. Yet interventions themselves also need interpretation.
This is methodological clarity rather than pessimism. Ask what is in the model and what it uses. Then ask what this intervention actually changed.
A fingerprint is worth examining without completing the chain of evidence alone. Patching and erasure make some counterfactuals executable, while still requiring attention to their targets, conditions and alternatives.
Hume's ghost never left. Whenever a familiar name or a clear heatmap makes understanding feel complete, it asks: which testable commitments does the explanation still owe?