softmax attention weights

Raw attention scores are messy — some big, some negative, none adding up to anything tidy. Softmax cleans them up. It takes each row of scores and squashes it into a set of positive weights that sum to exactly one, like slicing a pie among all the tokens. Now each weight reads directly as 'this fraction of my attention goes here,' and the token's new value is just that weighted average.

Softmax exponentiates every score and divides by the total, so the largest scores dominate while small ones shrink toward zero. The gap matters more than the absolute size: a score a few points higher than its neighbors can soak up most of the weight. This is also why scaling the scores beforehand matters — without it, large dot products would push softmax into a near-winner-take-all regime where gradients almost vanish.

One consequence is that attention is rarely perfectly sharp; some weight usually leaks onto many tokens, and models often park leftover weight on a single 'sink' token. Because every weight is positive, attention can only blend information additively — it cannot subtract.