RMS normalization (RMSNorm)

RMSNorm normalizes an activation vector by dividing it by its root-mean-square magnitude, then rescaling with a learned gain. Unlike LayerNorm it does not subtract the mean first, it fixes only the scale, not the center. Intuitively it answers how big the vector is and divides that out, leaving direction intact, which turns out to be the part of normalization that matters most for transformers.

For a vector x of dimension d, RMSNorm computes x times g divided by the square root of (one over d times the sum of x_i squared, plus epsilon), where g is a per-dimension learned scale and epsilon guards against division by zero. Dropping the mean-centering and the bias removes a reduction and a subtraction per token, making it cheaper and easier to implement efficiently, and empirically it matches or slightly beats LayerNorm in deep transformers. It is almost always used in the pre-norm position, applied to the residual stream before each sublayer.

RMSNorm is now the default normalizer in LLaMA, Mistral, and most modern LLMs, and its simplicity makes it friendlier to low-precision and fused kernels. The conceptual lesson is that re-centering, long assumed essential, is largely dispensable; controlling the activation scale is what stabilizes training.