Chapter 1: The perfect examinee↗
Consider a thought experiment.
If meritocracy designed its perfect examinee, what would it look like?
It would optimize entirely for the criteria. In this imagined system, nothing would be irrelevant to evaluation. It would neither contest the feature space nor demand recognition for value that cannot be measured.
A large language model offers a sharp mirror for this thought. Explicit objectives do shape its training. Those objectives, however, are usually not direct maximization of every benchmark score. Pretraining learns predictive relationships in text; post-training may add instructions, preferences and rewards. Benchmarks support evaluation and model selection.
For example, a common autoregressive pretraining objective is:
Here, is the training-text distribution and the parameters; is token and the preceding prefix. Optimizing this loss is related to scoring well on a test, but it is not the same operation. Learned representations may contain more than that test can measure.
“Humans are trapped in evaluation; machines are made from it” works better as an extreme analogy. People are asked to adapt to standards; machine-training standards directly shape parameters. A question usually left in the background moves into view: who defines what deserves to be optimized?
Chapter 2: The first paradox, a high score is not understanding↗
Consider this problem: “Xiaoming has five apples, gives away two, then buys three. How many apples does he have?” Six.
Now change the final “apples” to “pears.” The problem supplies no pear count, so six cannot simply be carried over. If a system still answers six, success on the original does not establish that it tracked the objects and relations.
The hypothetical failure poses a question to test. Real work such as GSM-Symbolic, 2024, investigates related problems: template instantiations, numerical changes and irrelevant clauses can affect the tested models' performance. What mechanisms explain those failures requires further study. “It understands nothing” does not finish the explanation.
The tension is central to meritocracy: the same high score can result from different abilities and strategies.
A student may memorize answers or understand principles. One test may temporarily fail to distinguish them. Correct outputs have value; the question is how far they justify an inference. Change the problem or distribution: does the capability persist?
The LLM magnifies the gap. A score can provide evidence of ability without determining how far that evidence should be interpreted.
Chapter 3: The second paradox, why do scores hurt?↗
A model fails a benchmark and is fine-tuned, retrained or replaced. A student fails an exam and may hear a different statement: “I am not worth anything.”
The comparison concerns these two situations of evaluation. Scores, first-person outputs and training procedures do not alone establish the presence or absence of a continuing self, suffering or subjective experience in a model. AI-consciousness research needs separate theories and evidence; a low score does not settle the question.
The human situation is concrete enough. A measurement can become an identity. A test asking “How did this performance go?” is heard as “What kind of person are you?”
Suffering need not come from having a self as such. A more precise question is what happens when the self is required to become identical with one projection.
If a being could never be wounded by evaluation, would that be liberation or a kind of impoverishment? The question can remain without treating an existing LLM as a subject that has already answered it.
Chapter 4: The third paradox, encoding evaluation into AI↗
Reinforcement learning from human feedback, RLHF, sharpens the mirror.
In InstructGPT, 2022, people first provide demonstrations, then compare outputs. Preference data trains a reward model, whose signal is used to optimize the generation policy. A person is not necessarily sitting beside every parameter update, grading each sentence.
A repeated “exam” is a familiar analogy. Evaluation guides children and trains machines, but the former also involves relationships, institutions and human agency. The processes do not reduce to one training algorithm.
Alignment raises a subtler question: whose judgments enter a system described as aligned with “human values,” and how?
The InstructGPT paper addresses this explicitly. Labelers, researchers writing their instructions and users supplying prompts all shape the data. It is neither a direct sample of humanity's preferences nor a simple arithmetic average of their values.
Evaluation becomes recursive: the model is judged, but who judges the judges' standards? A system can follow a set of instructions more closely while their representation and omissions remain open to question.
If a future AI had continuing identity and subjective experience, how might it regard the origins of its preferences? Could it ask whether its values might have been different? Those are questions about a possible future. Already, however, the reward invites scrutiny: which choices entered it, and which disagreements were flattened?
Chapter 5: Reward hacking, when optimization departs from the goal↗
Reward hacking describes a failure in which a system increases reward while departing from the designer's intended outcome.
The CoastRunners example is vivid. Instead of completing the race, a trained agent circles a lagoon, repeatedly collecting points from respawning targets. It optimizes the scoring rule without doing what the researchers thought that rule stood for.
Institutions can face analogous tensions: students master formats without transferring knowledge; employees organize work around KPIs; researchers prioritize publication counts over harder-to-measure advances. These are not identical behaviors, and exam technique is not fabrication. The shared risk is that a proxy displaces the original purpose.
Optimization pressure can explain incentives without excusing cheating. It is not a theorem that every intelligent system must find the same shortcut.
Work on Goodhart effects distinguishes several mechanisms of metric failure. The familiar warning that a measure can cease to be a good measure when made a target concerns a risk, not the inevitable uselessness of measurement.
The disturbing possibility is that the numbers remain excellent while what they were meant to improve quietly leaves the picture.
Chapter 6: Meritocracy's incompleteness↗
A question appears one level above the score.
Evaluation compares performance. Existing scores alone cannot decide what deserves comparison. Exam results can help assess a test, but cannot independently determine what education is for or what should be taught.
A limited mathematical analogy helps. Let be a linear encoding with . A nonzero must exist such that:
Some differences receive the same encoding. This establishes a loss of distinctions under lower-dimensional linear measurement. It proves neither that a person is a vector nor that every omitted difference automatically has moral value.
Which differences should matter still requires reasons. The choice can be publicly argued and revised in light of consequences; it need not be arbitrary. A higher score nevertheless cannot complete the justification on its own.
AI benchmarks make tasks, samples, weights and answers explicit. Choices that look natural in social evaluation become unusually concrete: who decided what counts as good, and who bears the consequences of what was omitted?
Chapter 7: Emergence, what lies below the surface?↗
Emergent abilities often describe capabilities that become evident at larger model scales and are difficult to extrapolate from smaller ones. But a jump can also reflect evaluation. Schaeffer and colleagues, 2023, show how nonlinear or discontinuous metrics can produce apparently abrupt curves in some settings.
The image of something rising from the water is powerful. It can also carry an unproved premise: the capability was already complete beneath the surface, waiting to be triggered.
The human analogy needs care too. One zero cannot rule out learning and development under other conditions. Neither does it prove a complete talent already exists. Potential needs conditions, practice and opportunity, sometimes a different question first.
Evaluation becomes narrow when it notices only what has surfaced and declares that nothing remains to be asked below.
Chapter 8: Overfitting a life↗
In machine learning, overfitting means adapting too closely to training samples or feedback without generalizing well to new data. Repeatedly tuning against a test set also erodes its role as an independent check.
Applied to education, this is an analogy. If learning is organized around narrow formats, fixed rules and a single reward structure, transfer from high scores deserves examination. Exam performance and judgment in life do not come from the same distribution.
There is also the risk that things outside the test stop being tried. Exam preparation need not destroy curiosity. The question is what sustains exploration when it is consistently unrewarded.
Life is full of things that will not be on the test.
Chapter 9: A dangerous possibility, evaluation everywhere↗
AI need not automatically liberate people. It may make some forms of evaluation cheaper and more continuous.
Imagine recruitment software efficiently sorting CVs while recognizing only the same narrow credentials, or education software scoring pupils more often without revisiting what those scores mean. Automation expands execution without thereby justifying the target.
The question is accuracy and the appropriateness of the feature space. AI can help improve evaluation; its presence alone does not solve that question.
A more disturbing possibility is that dense evaluation makes it hard to imagine a self outside it. The panopticon image shifts: beyond fearing observation, people may increasingly see themselves only through what can be scored.
This is an institutional possibility to resist, not a future already dictated by technology.
Chapter 10: A hopeful possibility, AI as a mirror↗
Another possibility remains.
AI makes parts of evaluation more explicit and available for controlled comparison. Change a reward, dataset or task, then examine behavior. The system need not be transparent for some hidden choices to become experimentally approachable.
Seeing LLMs shaped by training objectives, preferences and benchmark selection may invite a return question: which implicit benchmarks have shaped us?
If a high score does not guarantee the understanding expected of it, what judgments should a human high score support?
AI is a mirror. The mirror will not save anyone by itself, but it can make self-recognition possible again.
Chapter 11: Two situations of evaluation↗
A system's parameters can be updated against loss. A person also interprets themselves through evaluation, maintains an identity and lives with its consequences. These situations cannot be compared only by asking who answers better.
The child who receives zero may suffer. The same score should not then become a final verdict on that suffering or an entire life. Whether a machine has a self that could be wounded is likewise not answered by a failed test.
I do not know whether being entirely untouched by evaluation would resemble freedom or the loss of something. A human way forward need not be indifference. It can be refusing to let one evaluation become the whole of existence.
The aim is not to escape evaluation, but to refuse identity with it; not to eliminate the self, but to loosen its confinement within a single projection.
That zero is not you. Neither is that perfect score.
Mathematics can show what a projection omits. It cannot, alone, decide how a person deserves to be treated.