self-attention

Self-attention is attention turned inward: a sequence attending to itself. Instead of one sentence looking at a separate translation, every token in the very same passage looks at every other token in that passage, including itself. As the model processes 'bank' in 'she sat by the river bank,' self-attention lets that word gather signal from 'river' nearby and quietly settle on the geographic meaning rather than the financial one.

Mechanically, each token produces three vectors — a query, a key, and a value. The model compares one token's query against every token's key to get relevance scores, normalizes those into weights, and forms a weighted blend of the values. Every token does this at once, so a sequence of length n produces an n-by-n grid of interactions in a single pass. The output is a new sequence where each position has absorbed context from the whole.

The word 'self' simply distinguishes it from cross-attention, where queries come from one sequence and keys and values from another. In a decoder-only LLM, self-attention is the workhorse, repeated in every layer. It is also why memory and compute scale with the square of the sequence length — the central cost of long contexts.