1. A challenge from Berkeley↗
In 1980, the Berkeley philosopher John Searle published Minds, Brains, and Programs. His target was a position he called “strong AI”: running an appropriate program would give a computer genuine understanding and other mental states, and the program itself would explain human understanding.
Searle saw a conceptual leap. Does simulating a process also instantiate it?
A weather simulation does not make the room wet. Why should a simulation of a mind produce a real mind? The analogy puts a question before us, but cannot settle it alone. Distinguishing simulation from realization still requires knowing which properties understanding needs.
To bring the dispute into focus, he imagined the Chinese Room. Decades later, large models make its door feel close again.
2. The room↗
Imagine a room with a slot in its door. Inside sits a native English speaker who knows no Chinese. There are instructions in English, materials bearing Chinese symbols, paper and a pencil.
A note arrives: “What is your favourite colour?” The operator cannot read it, but can distinguish the shapes and follow instructions that produce another sequence: “I like blue.”
This is a simplified illustration of the procedure; Searle’s original example concerns stories and questions about them. Blue is incidental. The point is that the operator can work with symbols by their form without knowing what they refer to.
2.1 The test outside↗
More questions arrive: about yesterday, literature, artificial intelligence. Suppose the instructions and execution work so well that someone reading the answers cannot distinguish the room from a native Chinese speaker.
This is a stipulation of a thought experiment, not a room that has been built and tested. Nor must the instructions be an infinite dictionary listing every possible question and answer. A finite program can use memory and intermediate steps to process open-ended inputs.
A slightly more formal, simplified deterministic description is:
Here is the symbol input at step , the internal memory state, the output, and the update and output rule. This permits memory, computation and multiple steps, rather than mere lookup. The expression does not itself say whether those states have meaning.
3. Searle’s question↗
Now comes the question: does the operator understand Chinese?
By stipulation, no. The operator can return “blue” without knowing what it means, and need not even know that the symbols are questions and answers. Performing correctly and knowing what one is saying remain different things.
Searle then argues that a formal program alone is insufficient for understanding. Greater complexity or speed in symbol manipulation does not supply the missing semantics.
This is the argument’s forceful and contested step. The operator’s lack of understanding belongs to the setup; the whole system’s lack of understanding is a further conclusion that needs an argument. One cannot quietly stand in for the other.
4. Syntax and semantics↗
“Syntax” here means more than grammatical correctness: operations specified by form. Semantics concerns what the symbols mean.
Searle’s route can be summarized as follows. Programs are formally specified; minds have semantic content; formal operations alone do not constitute semantics. That final claim is a philosophical position he defends, not a computational theorem everyone already accepts.
Blue makes the tension visible. A system might classify colours, explain a term or correct its misuse. What seeing blue feels like is another question. Lacking a particular visual experience does not directly establish an inability to use colour concepts; using those concepts does not establish the same visual experience.
Calling an LLM “a larger Chinese Room” is therefore a provocative interpretation. It asks how linguistic competence relates to meaning. It does not settle the internal processes or experience of every model.
5. The counterattack↗
The Chinese Room became a meeting place for rival pictures of mind. Searle’s original article already considers these replies and his responses.
5.1 The systems reply↗
Perhaps the operator does not understand, but the system comprising operator, rules and memory does. A part’s lack of understanding does not establish that the whole lacks it.
Searle responds by having the operator memorize the instructions and materials, internalizing the entire system. The person still understands no Chinese, he maintains. A systems theorist can press further: why should that person’s judgment of their experience rule out a capacity at another level of organization?
The dispute concerns the level at which understanding belongs, not merely whether someone is sitting in the room.
5.2 The robot reply↗
Give the computer sensors and the capacity to act. Put it in the world. Symbols now connect not only to other symbols, but to things that can be perceived and acted upon.
Searle argues that if the operator still processes uninterpreted inputs and outputs, those connections have not explained how understanding arises. An opponent can ask whether describing everything as “still just symbols” leaves out the very changes in causal organization introduced by perception, learning and action.
A body does not automatically guarantee understanding. Reducing its contribution to an interface does not automatically establish that it contributes nothing.
5.3 The brain-simulator reply↗
What if the simulation reaches the level of neural activity? Searle imagines pipes and valves to argue that reproducing formal structure need not reproduce the causal powers responsible for intentionality.
“A simulated stomach cannot digest” does not settle this either. The physical processes needed for digestion are comparatively identifiable. Which causal properties understanding requires is precisely what is disputed. An analogy can expose a gap without filling it.
These exchanges clarify positions without dissolving the disagreement. Saying either that Searle has been conclusively refuted or that no effective reply has ever been made ends the debate on the reader’s behalf.
6. Decades later↗
Large models certainly differ from the imagined handbook. Many specific behaviours arise from learned parameters and representations, rather than individually scripted questions and answers. Distributed representations can reveal relations in vector space; Mikolov and colleagues’ work on word vectors is an early example of this approach.
But “the parameters were learned” does not mean “the system no longer runs a program.” Numerical operations on continuous-valued representations can be implemented computationally, and Searle’s challenge is not confined to hand-written lookup tables. Conversely, calling learned computations “just following rules” does not explain what those computations actually do.
Representations may be distributed or contain localized features. Not every concept must be spread across the entire network. A vector relation can help characterize a model’s organization without directly measuring human-like semantics or experience.
New architectures change the mechanisms that need investigation. A technical adjective does not retire the old question by itself.
7. Where meaning comes from↗
Searle emphasizes the causal powers responsible for intentionality, rather than a magical distinction between carbon and silicon. His original article explicitly allows the possibility of minds in artificial machines, including other chemical realizations. He rejects the sufficiency of program form alone.
Another route turns to how language is used. In Philosophical Investigations §43, Wittgenstein suggests that in a large class of cases, though not all, a word’s meaning can be explained through its use in the language.
“Blue” is not an isolated label. It describes skies and ink, and enters metaphor and emotional expression. Its meaning lies in relations within these practices; assigning each word a private mental picture cannot explain everything.
This makes the LLM’s position interesting. It learns many relations of use from human linguistic records. But are traces of use in text equivalent to participation in every practice? Can a system that explains a word also apply it, correct mistakes and maintain stable connections to its objects in new situations?
These questions can be separated and investigated. “Meaning as use” is not a pass certifying that an LLM understands.
8. Aboutness↗
One more term is needed: intentionality. It does not mean acting deliberately here, but the property of mental states being about something.
A thought of Paris is about Paris; fear of an exam is directed towards the exam. When a model writes “Paris,” in what way is its state about that city?
Searle distinguishes the intrinsic intentionality of mental states from the meaning that marks on a page derive from their users. On his account, a model’s symbols receive meaning through human interpretation. This is a theoretical position. The fact that outputs are symbols does not by itself establish that every internal state can have only derived meaning.
A mirror reflects human meaning without necessarily understanding what it reflects. The image captures a concern. Whether a model has only a mirror’s capacities still requires investigation; the metaphor cannot submit its answer for it.
9. Dennett’s intentional stance↗
Daniel Dennett approaches from another direction. In True Believers, he considers a strategy: treat a system as an agent with beliefs and desires, then use them to predict its actions.
“The thermostat wants to keep the temperature stable” can be a convenient description. For a complex system, the stronger question is whether such descriptions reliably predict a rich range of behaviour. Dennett is not merely suggesting that useful make-believe is acceptable. He connects the strategy’s success to the attribution of belief itself.
Along this route, saying that an LLM “believes” something should at least survive tests across questions and phrasings, rather than rest on a single self-report. Even a predictively useful description does not, on its own, establish subjective feeling.
In The Self as a Center of Narrative Gravity, Dennett also uses the centre of gravity to consider the self: it need not be a tiny thing concealed in the brain, but may be an abstract centre around which explanations are organized. Treating the self as an abstraction does not make human experience fake, nor establish that a machine has the same kind of self.
10. The impasse↗
These routes make understanding look less like a simple switch awaiting discovery.
Searle asks how a program could have intrinsic meaning. Attention to language use asks how understanding manifests in practice. Dennett asks which level of description reliably captures patterns of action. Part of their disagreement concerns what would count as an adequate explanation.
Everyday conversation does not wait for philosophy’s verdict. Talking to ChatGPT, it is easy to say “it thinks,” “it wants,” “it understands.” That response deserves attention. Sometimes it tracks a genuine behavioural capacity; sometimes it turns a fluent reply into a judgment about an inner mind.
The Chinese Room makes that leap visible. It does not thereby erase evidence of capabilities or reduce every judgment to illusion.
11. The ghost in the room↗
Look back at the room: an enclosed space, incoming signals, internal processing, outgoing actions. It readily calls to mind a brain inside a skull.
Similar outlines do not establish identical mechanisms. A brain maintains causal connections with its environment through a body; neural activity cannot simply be identified with the formal symbols in a handbook. Searle’s argument already centres on the brain’s causal properties, so this resemblance is not an objection he failed to notice.
A forceful question remains. If information always arrives through some transformation, what organization lets it be not only processed, but understood or experienced?
How can you know that you are not in a larger Chinese Room? The question can make an intuition pause without implying that human understanding has no evidence. Action, learning, correction and reports of experience all contribute evidence. Explaining how they fit together remains unfinished work.
The ghost in the room need not be another operator hiding behind the symbols. It might be an unfilled place in an explanation: behaviour has been described, but where do meaning and experience belong?
12. At the edge of the cliff↗
The preceding essay asked how compression and prediction could lead to understanding. The question now has another layer: does “understanding” mean a demonstrable capacity, a state with content, or the experience accompanying a process?
Capability, meaning and experience are related, but cannot stand in for one another in an argument. Research on model reasoning need not wait for a complete solution to consciousness. Explaining a function does not automatically settle experience either.
In Facing Up to the Problem of Consciousness, published in 1995, David Chalmers calls the existence of experience the “hard problem”: why should these processes be accompanied by something it is like to be that system? This is an influential philosophical distinction, not a final answer shared by every researcher.
The next essay turns to consciousness and embodied cognition: which capacities can a system demonstrate, and what remains to be explained about experience?