attention (intuition)

When you read 'the trophy did not fit in the suitcase because it was too big,' you instantly know that 'it' means the trophy, not the suitcase — you paid attention to the right earlier word. Attention is the mechanism that gives an LLM the same ability. For each word it is currently processing, the model decides how much to listen to every other word, and mixes them together accordingly.

The clever part is that this 'how much to listen' is learned and computed on the fly, as a set of weights that sum to one. A word forms a question about what it needs, every other word advertises what it offers, and the closeness between the two sets the weight. Highly relevant words get a large share; irrelevant words get almost none. The result is a fresh, context-aware version of each word.

This is why Transformers handle pronouns, long-distance agreement, and shifting topics so gracefully — there is no fixed window of 'nearby' words. Any token can reach back to any earlier token in a single step. The next few entries unpack exactly how those weights are built.