attention scores

Before a token can blend in context, the model has to rank how relevant every other token is. Those raw rankings are the attention scores: one number for each pair of tokens, saying how strongly the first should listen to the second. A high score means 'this word is important to me right now'; a low or negative score means 'mostly ignore it.'

Each score is a dot product between one token's query vector and another token's key vector — a measure of how aligned the two directions are. The scores are scaled down by the square root of the head dimension to stop them from getting so large that the next step saturates. For a sequence of n tokens you get an n-by-n table of these scores, one row per querying token, still raw and unnormalized at this stage.

On their own the scores are just similarities, not yet weights — they can be any size, positive or negative. The softmax step turns each row into a clean probability distribution. Scores are also where causal masking acts: future positions are set to negative infinity so they score zero after normalization.