positional encoding
/ puh-ZISH-un-ul en-KOH-ding /
Positional encoding is how a Transformer learns where each word sits, because attention on its own is blind to order. To self-attention, a sentence is an unordered bag of tokens — "dog bites man" and "man bites dog" would be indistinguishable. Positional encoding fixes this by stamping each token with information about its place in the line, so the model can tell first from second from last.
There are a few flavors. The original Transformer added fixed wave-like patterns — sines and cosines of different frequencies — to each token's vector, a clever scheme that needs no training and extends to lengths never seen. Many later models instead learn a position vector for each slot from data. Today's leading models often use rotary position embeddings (RoPE), which encode position by gently rotating the query and key vectors by an angle that depends on where the token is, so that what attention sees is the relative distance between tokens rather than an absolute address.
Positional encoding is small but load-bearing: remove it and a Transformer's grasp of grammar and meaning collapses, since word order is half of what language is. It also quietly shapes how far a model can reach. Schemes that encode absolute positions tend to break when asked to handle sequences longer than they were trained on, while relative and rotary schemes generalize better — which is one reason the method you choose has a direct effect on a model's usable context length.
Two sentences use the same words: "only she told him" versus "she told only him." Self-attention alone can't separate them. Positional encoding gives each "only" a different position stamp, so the model knows which word it limits.
Same words, different order — only the position stamp tells them apart.
How position is encoded directly limits how far a model can read. Absolute schemes often degrade sharply past their trained length; relative and rotary schemes extend more gracefully — but none is magic, and "long-context" claims should always be checked against measured accuracy at that length, not just the advertised number.