batch normalization

Deep networks are hard to train partly because the distribution of values flowing into each layer keeps shifting as the layers below it update — a moving target that forces small, timid learning rates. Batch normalization (Ioffe and Szegedy, 2015) attacks this by standardizing the inputs to a layer on the fly: for each feature, it subtracts the mean and divides by the standard deviation computed over the current mini-batch, so every feature arrives roughly zero-mean and unit-variance. The payoff is dramatic — much faster, more stable training, tolerance for higher learning rates, and reduced sensitivity to how the weights were initialized. It was a key enabler of very deep CNNs like ResNet.

Mechanically, for a feature x over a mini-batch, BN computes the batch mean μ_B and variance σ²_B, normalizes x̂ = (x − μ_B) / √(σ²_B + ε), then applies a learnable scale γ and shift β to produce y = γ x̂ + β. The crucial detail is that γ and β are trainable parameters: pure normalization would rob the layer of the ability to represent non-zero-mean or large-variance features, so BN lets the network learn to undo the normalization where useful — it can even recover the identity if that is best. In CNNs, the statistics are computed per channel, pooled over the batch and all spatial positions, so one γ and β are shared across an entire feature map.

Why it helps was originally attributed to reducing 'internal covariate shift' (the shifting input distributions), but later work (Santurkar et al.) argued the deeper reason is that BN smooths the loss landscape, making gradients more predictable and allowing larger steps. Whatever the mechanism, BN also injects a mild, useful regularization: because each example's normalization depends on the random other members of its batch, the network sees slightly noisy activations, similar in spirit to dropout.

BN has two notorious sharp edges. First, it behaves differently at training and inference time: during training it uses the current batch's statistics, but at inference there may be no batch, so it uses running averages of the mean and variance accumulated during training. Forgetting to switch the model to eval mode is a classic bug that silently wrecks accuracy. Second, BN degrades when the batch is very small (statistics become unreliable) or when batches are not i.i.d. across devices, which is why small-batch detection and segmentation often replace it with group normalization or use synchronized BN, and why transformers use layer normalization instead.