positional encoding

Attention has a blind spot: by itself it treats a sentence as a bag of words. 'Dog bites man' and 'man bites dog' would look identical, because attention compares tokens by content, not by where they sit. Positional encoding fixes this by stamping each token with information about its place in the sequence, so the model can tell first from last and near from far.

Early Transformers added fixed sine-and-cosine patterns of different frequencies to the token embeddings, giving every position a unique fingerprint. Today most LLMs use rotary position embeddings (RoPE), which rotate the query and key vectors by an angle that depends on position; the dot product between two tokens then naturally encodes how far apart they are. Other schemes, like ALiBi, instead add a distance-based penalty straight onto the attention scores.

How positions are encoded strongly shapes how far a model can generalize. Schemes that depend on relative distance tend to stretch to longer contexts more gracefully, and tricks that rescale RoPE's frequencies are a common way to extend a model's window past what it was trained on. Position handling is quietly one of the most active areas of LLM engineering.