scaled dot-product attention

Scaled dot-product attention is the exact arithmetic at the heart of every transformer. It answers one question for each token: given my query, how much should I take from each other token's value? The recipe has three steps — measure similarity, turn similarities into a probability-like weighting, and take a weighted sum — plus one small but essential correction factor, the scaling, that keeps the whole thing trainable.

Step one: similarity is measured by a dot product. The query vector of a token is multiplied (inner product) with the key vector of every token; a larger dot product means the two vectors point in more similar directions, i.e. higher relevance. Stacking all queries and keys into matrices Q and K, all the scores at once are the matrix product Q times K-transpose. Step two: each row of scores is passed through a softmax, which exponentiates and normalizes them into non-negative weights summing to one — a soft selection over tokens. Step three: these weights multiply the value matrix V, yielding for each token a weighted average of all values. In one line: Attention(Q,K,V) = softmax(QKᵀ / √d_k) · V.

The √d_k division is the 'scaled' part and it is not cosmetic. When the key dimension d_k is large, dot products of random query/key vectors have variance proportional to d_k, so their magnitudes grow; feeding very large numbers into softmax pushes it into a near one-hot regime where its gradient is almost zero, stalling learning. Dividing each score by the square root of d_k normalizes the variance back to order one, keeping softmax in a responsive range. This single factor is what makes deep attention stacks train stably.

The dominant cost lives in QKᵀ and in applying the weights to V: both are O(N² · d) for a sequence of N tokens of width d, and the N x N score matrix is also O(N²) memory. For high-resolution images N is large, so this quadratic term is the bottleneck that motivates windowed attention (Swin), linear-attention approximations, and IO-aware exact kernels such as FlashAttention that avoid materializing the full score matrix.