self-attention

Self-attention is the mechanism that lets every token in a sequence directly consult every other token and update itself based on what it finds relevant. Picture a roundtable where each patch can ask the whole room a question, listen to everyone's answer weighted by how relevant they are, and rewrite its own notes accordingly. 'Self' means the questions, the candidates being matched, and the information retrieved all come from the same sequence — the image is talking to itself. This is what gives a transformer a global receptive field from the very first layer: a patch in one corner can be influenced by a patch in the opposite corner immediately.

It works through three learned views of each token. Every token is projected into a query (what I am looking for), a key (what I offer to be matched against), and a value (the information I will hand over if matched). For a given token, you compare its query against the keys of all tokens to get a relevance score for each, normalize those scores into weights with a softmax so they are positive and sum to one, and then take the weighted average of all the values. That weighted average is the token's new representation — a blend of everyone else's information, dominated by whoever was most relevant.

Two properties define its character. First, it is content-based and dynamic: the weights are computed from the data at runtime, so which patches talk to which changes per image, unlike a convolution's fixed receptive window. Second, it is permutation-equivariant: with no positional encoding, reordering the input merely reorders the output, which is exactly why positions must be injected. The price is quadratic cost — comparing every token to every other is O(N²) in the number of tokens N — which is the central scaling pain of long sequences and high-resolution images.

Self-attention contrasts with cross-attention, where the queries come from one sequence and the keys/values from another (as when a decoder attends to an encoder, or a text prompt attends to image features). The same arithmetic underlies both; 'self' simply names the case where all three projections read the same input. In a vision transformer, self-attention is the only operation that mixes information across spatial locations — the MLP that follows it works on each token independently.