self-attention
/ self uh-TEN-shun /
Self-attention is attention turned inward: every word in a sequence looks at every other word in the same sequence — including itself — to build a richer sense of its own meaning. Where ordinary attention might let a translation's output peek back at the input, self-attention has the input examine itself. Each position asks, "Given everyone else here, what should I really mean in this context?"
Mechanically, each word produces three things from its own vector: a query (what am I looking for?), a key (what do I offer?), and a value (what content do I pass along?). A word's query is compared against every word's key to get relevance scores; those scores become weights, and the word collects a weighted blend of everyone's values. The upshot is that the representation of "it" can absorb information from "trophy," the representation of "bank" can absorb information from "river," and so on — all computed in parallel, in a single pass, for every position at once.
This is the engine room of the Transformer. Because all positions attend simultaneously rather than marching left to right, self-attention is highly parallel and trains fast on modern hardware, and it lets distant words influence each other directly instead of through a long chain of intermediate steps. The honest cost is scale: comparing every word with every other word means the work grows with the square of the sequence length, which is precisely why very long inputs remain expensive and why so much engineering effort goes into taming that cost.
In "The animal didn't cross the street because it was too tired," self-attention lets the vector for "it" gather most of its meaning from "animal" (not "street"). Swap "tired" for "wide" and the same mechanism shifts "it" toward "street" instead.
Each word's meaning is re-mixed from the whole sentence, all at once.
Self-attention has no built-in sense of word order — to it, a sentence is a bag of items. "Dog bites man" and "man bites dog" would look identical without help. That help comes from positional encoding, which is why the two ideas always travel together.