Attention: What Is the Model Looking At?

Chapter 1: Descartes' puzzle↗

When looking at a piece of wax, what exactly is being seen?

In the Second Meditation, published in 1641, Descartes brings a piece of wax, fresh from a beehive, near a fire. Its color, shape, smell and texture change, yet it is recognized as the same wax.

Seeing becomes more than an inventory of sensory properties. Descartes appeals to an inspection of the mind, inspectio mentis, to discuss understanding and judgment. This is not yet a computational theory of selective attention.

More than three centuries later, a related question appears in neural networks: what is meant when a model is said to “attend”?

Chapter 2: A counterintuitive turn↗

Recurrent models such as RNNs, LSTMs and GRUs update a state along the sequence. The next state depends on the previous one. Information resembles water passing through successive gates.

That feels natural. Language unfolds in order; music unfolds in time. But ordered inputs do not require every internal computation to queue in the same order.

Attention did not begin in 2017. Bahdanau, Cho and Bengio's work, released in 2014 and published at ICLR 2015, already let translation models read different source positions as needed, alleviating the bottleneck of a fixed-length summary.

In 2017, Attention Is All You Need introduced the Transformer, replacing the central sequence-aligned recurrent path with self-attention. Positions could interact directly rather than passing information through every intermediate state.

That interaction has a visibility boundary. Full encoder self-attention can read the entire sentence; an autoregressive decoder uses a causal mask to exclude future information. Training can process many positions of a known sequence in parallel, while generation still usually produces new tokens one at a time.

The organization of computational dependencies changes. Time and order do not disappear.

Chapter 3: An analogy, the detective's inquiry↗

Imagine a detective investigating a case.

Along the recurrent route, witnesses are interviewed in order and earlier statements continually folded into notes. By the hundredth witness, details from the first may be hard to recover. This illustrates transmission through a limited state; recurrent systems with attention need not rely on a single summary alone.

Self-attention is more like placing statements in one hall, where each position can read other available material as needed. In causal decoding, later statements have not been released. Earlier positions cannot peek ahead.

Which arrangement is more likely to uncover the truth? That depends on the task, representation and training. A hall shortens communication paths without guaranteeing accurate testimony or a correct investigation.

Chapter 4: The mathematics of attention↗

Open the hood. What actually happens in that hall?

4.1 Query, key and value: search, matching and content↗

Start with three roles:

• Query: what conditions does this position use to seek information?
• Key: what vector does another position offer for matching?
• Value: what content is actually transmitted after matching?

Think of entering a library with a query about quantum mechanics, matching catalogue information, then reading the contents. Model Q, K and V vectors are not hand-written topic labels, and the operation need not select just one book.

For self-attention, let $X\in\mathbb{R}^{n\times d}$ contain representations at $n$ positions. Learned matrices produce:

Here, $Q,K\in\mathbb{R}^{n\times d_k}$ and $V\in\mathbb{R}^{n\times d_v}$. Different projections give the same input its query, key and value roles. Cross-attention can obtain queries and key–value pairs from different sequences.

4.2 Scaled dot products: allocate weights, then collect content↗

Basic scaled dot-product attention is:

The mask $M\in\mathbb{R}^{n\times n}$ adds zero for visible positions and $-\infty$ for prohibited ones. Row $i$ of $A$ distributes query position $i$'s weights across keys; $O\in\mathbb{R}^{n\times d_v}$ contains the aggregated outputs.

First, $QK^\top$ gives compatibility scores. A dot product depends on vector lengths as well as directions. It is neither simply cosine similarity nor a statistical correlation coefficient.

Second, divide by $\sqrt{d_k}$ and apply softmax row by row. In this basic form, each row has nonnegative weights summing to one; masked positions receive no weight.

Third, combine values with those weights. Before output projection, a head produces a weighted combination rather than retrieving one untouched book. Attention dropout or a subsequent output projection also prevents treating the entire module as that same convex combination.

4.3 Why divide by a square root?↗

The original paper offers an illustrative assumption. If all components of $q$ and $k$ are mutually independent, with mean zero and variance one, then $q^\top k$ has variance $d_k$. Larger dimensions can produce large scores, pushing softmax toward sharp distributions with very small gradients. Scaling restores variance one under this assumption.

This motivates a design choice. It does not mean trained representations always behave as independent random variables. The weights provide learned information routing, not an explanation that is valid by default.

Chapter 5: Multiple heads, multiple perspectives↗

One head may not suffice because matching need not follow one criterion.

Back in the library, one reader cares about topic, another about difficulty, a third about date. This offers an intuition for different projections; it does not assign those jobs to heads in advance.

Multi-head attention computes with separate $W_r^Q,W_r^K,W_r^V$ and combines the results. If head $r$ outputs $O^{(r)}$, write:

After concatenation, $Z\in\mathbb{R}^{n\times h d_v}$, with output projection $W^O\in\mathbb{R}^{h d_v\times d}$. The original base Transformer uses eight heads, $d=512$, and $d_k=d_v=64$ per head. These are configuration choices, not constants for every model.

Clark and colleagues' 2019 analysis of BERT finds heads attending to delimiters, particular positional offsets, or patterns corresponding to syntax and coreference. Heads need not be nonredundant, and a pattern alone does not show that one head independently performs a linguistic function.

A detective team can read clues differently. Establishing who advanced the conclusion also requires following the transmitted content and its downstream effects.

Chapter 6: Positional encoding, the ghost of the sequence↗

Wait. If positions match directly, where does order enter?

6.1 The same words, a different relation↗

Consider a deterministic full self-attention map $F$ without positional encoding or masking. If $P$ permutes input rows, then:

Permute the input, and the output permutes with it. This is permutation equivariance, not an unchanged output matrix. For corresponding words, however, this layer alone does not identify their original sequence indices.

The Chinese sentences “我爱你” and “你爱我” contain the same words—“I,” “love” and “you”—but express different relations. Without an additional order signal, full matching cannot be expected to know which came first.

A causal mask changes the premise: different positions see different prefixes, introducing asymmetry. The unmasked equivariance argument does not apply unchanged to every Transformer setting.

6.2 Sinusoidal positions↗

The original paper adds positional encodings to input embeddings. Its sinusoidal version is:

Here, $p$ is position and $i=0,\ldots,d/2-1$ indexes sine–cosine dimension pairs, assuming even $d$. Each pair has a different frequency. For a fixed offset, the new encoding within each pair is a linear transformation of the old one.

The same word embedding at the first and third positions therefore becomes different after position is added.

These functions can be evaluated beyond the training length. That does not guarantee reliable behavior on unseen long sequences. Computing a new coordinate and learning to use it are different achievements.

Chapter 7: Attention as epistemology↗

Return to Descartes.

Why is wax recognized as the same wax after its appearance changes? The question makes the organization and judgment within seeing less self-evident. A Transformer offers a specific computation: calculate content-dependent matching weights, aggregate vectors and pass the result onward.

There is a suggestive connection, not a philosophical proof supplied by the formula. Content-dependent weighting does not establish mental inspection, understanding or a subjective gaze in Descartes' sense.

An attention visualization does show part of the model's computation. How that part contributes to the result cannot be read from heatmap colors alone. Jain and Wallace, 2019, show that different weights can yield similar predictions in some settings; this does not prove attention has no causal role. For the methodological debate, see Hume's Ghost.

Calling a machine “attentive” first identifies a computational relationship. What psychological meaning is being added beyond it?

That may be the least familiar question inside this familiar word.

第一章：笛卡尔的困惑↗

当看着一块蜡时，到底在“看”什么？

在1641年发表的《第二沉思》中，笛卡尔把一块刚从蜂巢取出的蜡放近火旁。颜色、形状、气味与触感改变了，却仍被认作同一块蜡。

这使“看见”不再只是感官性质的清单。笛卡尔诉诸心智的审视（inspectio mentis），讨论理解与判断如何参与对蜡的认识；这还不是一套关于选择性注意的计算理论。

三百多年后，一个相近的疑问在神经网络中出现：当说模型在“注意”时，究竟指什么？

第二章：一个反直觉的转向↗

RNN、LSTM、GRU等递归模型沿序列更新状态：要得到下一步的状态，先要有上一步的状态。信息像水流，逐个经过闸门。

这很自然。语言按顺序展开，音乐也在时间中发生。但输入有顺序，不意味着所有内部计算都必须按同一种顺序排队。

注意力也不是2017年才出现。Bahdanau、Cho与Bengio在2014年公开、发表于ICLR 2015的工作，已经让翻译模型根据当前需要，从源句不同位置读取信息，减轻固定长度摘要的瓶颈。

2017年的Attention Is All You Need进一步提出Transformer，用自注意力代替沿序列位置递归传递状态的核心路径。位置之间可以直接建立联系，不必让信息穿过每一个中间状态。

但“直接联系”有可见范围：编码器的全连接自注意力可以读取整句；自回归解码器则用因果遮罩限制未来信息。训练时可并行计算已知序列中的许多位置，生成时仍然通常逐个产生新token。

改变的是计算依赖的组织方式，不是时间与顺序被取消了。

第三章：一个类比，侦探的询问↗

想象一位侦探调查案件。

沿递归路线，他按次序询问证人，把先前内容不断更新进笔记。到第100位证人时，第一位的细节可能已经难以从摘要中找回。这是在比喻有限状态的信息传递，不代表所有带注意力的递归系统都只剩一本摘要。

自注意力更像把证词放进同一个大厅：一个位置可以按需要读取其他可见位置的材料。若是因果解码，后面的证词尚未开放，前面的人也不能偷看。

哪种安排更容易发现真相？这取决于任务、表示与训练。大厅让交流路径变短，却不保证证词准确，也不保证询问已经找到了真相。

第四章：注意力的数学↗

打开引擎盖，看看这个大厅里实际发生了什么。

4.1 Query、Key、Value：查询、匹配与内容↗

三个名字可以先这样理解：

• Query：这个位置用什么条件寻找信息？
• Key：其他位置以什么向量参与匹配？
• Value：匹配之后，实际传来的内容是什么？

像走进图书馆，带着“量子力学”的查询，用目录信息匹配书籍，再读取内容。但模型里的Q、K、V不是人工写好的主题标签，也不一定只挑出一本书。

对自注意力，设$X\in\mathbb{R}^{n\times d}$包含$n$个位置的表示，通过学习到的矩阵得到：

$Q,K\in\mathbb{R}^{n\times d_k}$，$V\in\mathbb{R}^{n\times d_v}$。同一输入由不同投影承担查询、键与值的角色；跨注意力则可以从不同序列取得查询与键值。

4.2 缩放点积：先分配权重，再汇集内容↗

基本的缩放点积注意力可以写成：

$M\in\mathbb{R}^{n\times n}$表示遮罩：可见位置加0，禁止位置加$-\infty$。$A$的第$i$行，是查询位置$i$分配给各个键的权重；$O\in\mathbb{R}^{n\times d_v}$是汇集后的输出。

第一步，$QK^\top$给出匹配分数。点积同时依赖方向与向量长度，不能直接当成余弦相似度，更不是统计学上的相关系数。

第二步，除以$\sqrt{d_k}$，再逐行做softmax。在这个基本形式中，每行权重非负且和为1；遮住的位置不会获得权重。

第三步，用这些权重汇集Value。单个头在输出投影前得到加权组合，而不是从书架上原样取出唯一一本书。加入attention dropout或后续输出投影后，也不能把整个模块继续当成这同一个凸组合。

4.3 为什么除以平方根？↗

原论文给出一个说明性假设：若$q$与$k$的各分量相互独立、均值为0且方差为1，那么$q^\top k$的方差为$d_k$。维度增加时，分数容易变得很大，使softmax趋于尖锐、梯度很小；缩放把这个假设下的方差拉回1。

这解释了一种设计动机，不是说训练后的神经表示始终满足独立随机变量假设。权重是一种可学习的信息路由，不是一份天然成立的解释。

第五章：多头注意力，视角的多元性↗

一个头可能不够，因为匹配关系不必只有一种。

回到图书馆：一个读者关心主题，另一个关心难度，还有一个关心年代。这是理解多种投影的直觉，并不是每个头都已经被分配了这样的职责。

多头注意力用不同的$W_r^Q,W_r^K,W_r^V$分别计算，再合并。若第$r$个头输出为$O^{(r)}$，可写成：

拼接后$Z\in\mathbb{R}^{n\times h d_v}$，输出投影$W^O\in\mathbb{R}^{h d_v\times d}$。原始Transformer的base配置使用8个头，$d=512$，每个头$d_k=d_v=64$；这不是所有模型固定不变的数字。

Clark等人对BERT的分析（2019）观察到，某些头偏向分隔符、特定位置距离，或与语法和共指关系相合。它们并非都互不重复，也不能仅凭图样就断定一个头独自完成了某项语言功能。

侦探团队可以有不同读法，但要判断谁真正推动了结论，还需要追踪传来的内容，以及它对后续计算的影响。

第六章：位置编码，序列的幽灵↗

等等，如果位置之间直接匹配，顺序信息在哪里？

6.1 “我爱你”与“你爱我”↗

考虑推理时一个没有位置编码、没有遮罩的全连接自注意力映射$F$。如果$P$是重排输入行的置换矩阵，那么：

也就是说，重排输入，输出会以同样方式重排。这叫置换等变，不是输出矩阵完全不变。但对相应的词而言，单靠这一层无法知道自己原先位于第几个位置。

“我爱你”与“你爱我”含有同一组词，却有不同的关系。若没有进一步的顺序信号，不能指望这层全连接匹配自己知道谁在前、谁在后。

因果遮罩会改变上述前提：它让不同位置看到不同前缀，本身就带来不对称。不能把无遮挡情形的等变性原样套到所有Transformer设置中。

6.2 正弦位置编码↗

原论文把位置编码加到输入嵌入中。其正弦版本为：

$p$是位置，$i=0,\ldots,d/2-1$索引正弦／余弦维度对，假定$d$为偶数。不同维度使用不同频率；对固定偏移，同一频率对的新位置编码可由旧位置编码作线性变换得到。

于是，“我”在第一个位置与第三个位置，词嵌入即使相同，与位置编码相加后也不同。

这些函数可以在训练长度之外计算，却不保证模型在未见过的长序列上仍然可靠。能写出一个新位置的坐标，与学会怎样使用它，是两件事。

第七章：注意力作为认识论↗

回到笛卡尔。

蜡的外观改变，为什么仍被认识为同一块蜡？这个问题提醒，“看见”里包含着怎样的组织与判断。Transformer提供的则是一种明确计算：根据输入内容计算匹配权重，汇集向量，再交给后续层。

二者之间有启发性的呼应，却没有由公式直接连上的哲学证明。内容依赖的加权，不足以证明模型拥有笛卡尔所说的心智审视、理解或主观注视。

当可视化注意力时，看见的确实是模型计算的一部分。但那部分怎样参与结果，不能只靠热图颜色来回答。Jain与Wallace（2019）展示了某些设置下不同权重可以产生近似相同预测；这不等于证明注意力没有因果作用。相关的方法争论，可继续读《休谟的幽灵》。

说机器在“注意”，首先是在指认一种计算关系。 在此之外，还赋予了它什么心理含义？

这也许是这个熟悉的词，留给解释者最不熟悉的问题。