layer normalization

Batch normalization standardizes each feature across the examples in a batch, which ties every example's processing to its batchmates and breaks down when the batch is tiny or absent. Layer normalization (Ba, Kiros, and Hinton, 2016) makes the opposite cut: it standardizes across the features within a single example, independently of every other example. Picture the activations of one image as a row of numbers — LayerNorm rescales that one row to zero mean and unit variance on its own, with no reference to any other image. This makes it completely batch-independent: it behaves identically whether the batch has one example or a thousand, and identically at training and inference time.

Formally, for one example with a feature vector x of dimension d, LayerNorm computes the mean μ and variance σ² over those d features, normalizes x̂ = (x − μ) / √(σ² + ε), and applies a learnable scale γ and shift β (each of dimension d) to give y = γ ⊙ x̂ + β, where ⊙ is elementwise multiplication. The mean and variance are over the feature dimension of that single token or position, so no statistics ever cross between examples — the property that makes it robust to small batches and to variable-length sequences.

This batch-independence is why layer normalization is the standard normalizer in transformers, and therefore in vision transformers (ViT), DETR, CLIP's text and image encoders, and most modern multimodal models. Transformers process sets of tokens of varying number, often with tiny effective batches per device, exactly the regime where batch normalization struggles. A practical wrinkle is placement: 'Post-LN' (normalize after the residual addition, as in the original transformer) can be unstable to train at depth and needs careful warmup, whereas 'Pre-LN' (normalize the input to each sublayer, before attention or the MLP) gives much more stable gradients and is now the default in deep vision and language transformers.

A widely-used simplification is RMSNorm, which drops the mean-subtraction and the bias β and divides only by the root-mean-square of the features, y = γ ⊙ x / RMS(x). It is cheaper and empirically about as effective, and now appears in many large models. The broader point is that BN, LN, GroupNorm, and InstanceNorm are all the same recipe — normalize, then learn a scale and shift — differing only in which axes they pool the statistics over, and you choose among them based on architecture and batch regime.