layer normalization

Deep networks are touchy: as signals pass through dozens of layers, their scale can balloon or collapse, and training stalls. Layer normalization keeps things steady. Before each sublayer, it takes a token's vector and rescales it so its values have a consistent size and spread — roughly, re-centering and re-sizing the numbers so the next layer always receives input in a familiar range.

For each token independently, layer norm subtracts the mean and divides by the standard deviation across that token's features, then applies a learned scale and shift. Unlike batch normalization, it never looks across other examples, which makes it well suited to variable-length text and small batches. Most modern LLMs place it before the sublayer ('pre-norm'), which makes deep stacks far more stable to train; a common cheaper variant, RMSNorm, skips the mean and just rescales.

Normalization is quiet but load-bearing. Move it to the wrong spot, or remove it, and large models often fail to train at all. It works hand in hand with the residual stream: the stream carries the running sum, and layer norm controls the scale at which each block reads from that sum.