RMSNorm

Deep stacks of layers can let activations blow up or shrink toward zero, so every block first rescales its input to a sane size. RMSNorm does this with the lightest possible touch: it divides each vector by its own root-mean-square magnitude, then multiplies by a learned per-feature gain. Think of it as a volume knob that resets the loudness of every token's representation to a fixed level before the layer does its work, without caring about the average value.

It is a stripped-down LayerNorm. Classic LayerNorm subtracts the mean and divides by the standard deviation; RMSNorm drops the mean-subtraction and the bias entirely, keeping only the scaling. In practice that removed step barely affects quality but saves computation and is numerically simpler, which is why most current LLMs use RMSNorm. It is paired with the placement question: nearly all modern models put the norm before the attention and feed-forward sub-layers rather than after.