scaled dot-product attention

Scaled dot-product attention is the exact recipe most Transformers use to turn queries, keys, and values into output. It has four steps, and each one earns its place. First, score every query against every key with a dot product — a single number measuring how aligned two vectors are. Second, divide those scores by the square root of the key's dimension; this is the "scaled" part. Third, apply a softmax so the scores become positive weights summing to one. Fourth, take the weighted sum of the values.

Why bother dividing? Because the dot product of two long vectors tends to grow large simply because there are many terms being added. If the raw scores get big, softmax pushes nearly all the weight onto a single item and flattens everything else to zero — and at that extreme the gradients used for learning shrink toward nothing, so training stalls. Dividing by the square root of the dimension keeps the scores at a sane size so the softmax stays soft and the model keeps learning. It is a small numerical fix with outsized practical importance.

The whole thing is just one line of linear algebra: softmax of (Q times K-transpose, divided by the square root of d) times V. That compactness is the point — it is cheap, fully parallel, and differentiable, so it slots neatly into hardware that loves matrix multiplication. The honest caveat is the same one that haunts all attention: that Q-times-K step builds a score for every pair of positions, so memory and compute scale with the square of sequence length, which is exactly the wall that flash attention and other tricks were invented to push back.