ALiBi (attention with linear biases)

ALiBi removes positional embeddings entirely and instead biases the attention scores by how far apart two tokens are. Before the softmax, it subtracts a penalty proportional to the distance between query and key: the further back a token sits, the more its score is discounted. There is no learned position vector at all, position lives purely in this distance penalty.

Concretely, for a query at position i and a key at position j with j at most i, ALiBi adds minus m times (i minus j) to the pre-softmax score, where m is a fixed head-specific slope. Each head gets a different slope drawn from a geometric sequence, so some heads attend very locally with a steep slope while others attend almost globally with a shallow one. Because the bias is a simple linear function of distance, untied to any trained range, a model trained at length 1024 can be evaluated at much longer lengths with little degradation.

ALiBi's headline property is length extrapolation: train short, test long. Its weakness is that the monotone recency penalty bakes in a strong locality prior, which can hurt tasks needing sharp long-range retrieval; RoPE with frequency rescaling such as YaRN has largely become the more popular route to long context, but ALiBi remains influential and appears in models like BLOOM and MPT.