attention mechanism
/ uh-TEN-shun MEK-uh-niz-um /
An attention mechanism is a way for a model to decide, for each thing it is currently working on, which other pieces of information to lean on most heavily. Picture reading the sentence "The trophy didn't fit in the suitcase because it was too big." To figure out what "it" refers to, your eyes flick back to "trophy" and "suitcase" and weigh them. Attention is the machine's version of that flick: instead of squashing a whole sentence into one fixed summary, the model keeps every word available and, at each step, draws a weighted blend of the ones that matter.
Concretely, attention turns "how relevant is item A to item B?" into a number. The model computes a score between the thing it is focused on and every other item, passes those scores through a softmax so they become positive weights that add up to one, and then takes a weighted average of the items' contents. Words with high weight contribute a lot; words with near-zero weight are effectively ignored. Nothing is thrown away in advance — the relevance is recomputed fresh for each position.
Attention mattered because the older approach forced a model to compress an entire input into a single bottleneck vector before producing any output, and long inputs got mangled in that squeeze. Attention removed the bottleneck: the output can reach back to any part of the input directly. It is worth being precise, though — attention is a weighted-averaging trick, not understanding or reasoning. The weights tell you where the model looked, not why, and reading them as a tidy explanation of the model's logic is a known overreach.
Translating "the bank of the river" into French, the model must choose between rive (riverbank) and banque (money bank). When it reaches "bank," attention puts heavy weight on the nearby word "river," and that context tips it toward rive.
Attention lets a word's meaning be resolved by glancing at the words around it.
"Where the model attended" is tempting to read as "why the model decided," but they are not the same. Attention weights are one ingredient among many, and studies show you can often change them substantially without changing the output — so treat attention maps as a clue, not a confession.