positional encoding

Positional encoding is how a transformer learns where each token sits. Self-attention has a strange property: it treats its input as a bag, not a line. If you shuffle the order of the tokens, attention produces the same set of outputs, just shuffled the same way — it cannot tell which patch was top-left and which was bottom-right. For language that would scramble word order; for images it would scramble layout. Positional encoding fixes this by stamping each token with a signature of its location before attention ever runs.

The mechanism is simple addition. To each token's embedding you add a position vector of the same dimension D, so token at position i carries embedding plus pos(i). Two main flavours exist. Sinusoidal encodings (from the original transformer) define pos(i) analytically as sines and cosines of i at geometrically spaced frequencies; they need no training and extrapolate smoothly to longer sequences. Learned position embeddings (used by the original ViT) make pos(i) a free parameter vector for each slot, trained by backprop; they are flexible but tied to the sequence length seen in training.

For images, 'position' is two-dimensional, and several refinements follow. You can give separate learned embeddings for the row and the column and add them (factorized 2D), or use 2D sinusoids. More importantly, modern vision transformers increasingly favour relative position information: instead of encoding absolute coordinates, they bias each attention score by a learned function of the offset between the two tokens (Swin's relative position bias) or rotate queries and keys by a position-dependent angle (rotary embeddings, RoPE). Relative schemes capture the intuition that 'two patches three steps apart' should behave the same wherever they are in the image.

Why it matters in practice: because the plain ViT uses learned absolute embeddings indexed by patch slot, feeding a different image resolution changes the number of patches and breaks the table. The standard fix is to interpolate the position embeddings (treat them as a small 2D grid and resize it) when fine-tuning at higher resolution — a step you must remember, or accuracy collapses.