batch normalization

Batch normalization is a housekeeping step inside a deep network that keeps the numbers flowing between layers from drifting to wild scales. As data passes through layer after layer, the values can balloon or shrink unpredictably, which makes training slow and twitchy — a bit like trying to have a steady conversation in a room where the volume keeps lurching from a whisper to a shout. Batch norm steadies that volume so every layer receives input in a sane, consistent range.

Concretely, training happens in small groups of examples called batches (a handful or a few hundred at a time). For each batch, batch norm looks at the values arriving at a layer, subtracts their average and divides by their spread, so they come out centered around zero with a tidy scale. It then multiplies by two learned numbers — a scale and a shift — so the network can, if it wants, undo the normalization where that turns out to be better. This rescaling-on-the-fly lets you train faster, use larger learning rates, and worry less about exactly how the weights were initialized.

There are honest catches. Because the normalization uses statistics of the whole batch, it behaves oddly when batches are tiny, and it needs separate bookkeeping at inference time (when you may feed one example at a time, using a running average gathered during training). It also fits awkwardly with sequence models, where the batch is not the natural thing to normalize over — which is exactly why layer normalization was invented as an alternative. Even so, batch norm was a turning point: after its introduction in 2015, deep networks suddenly became markedly easier and faster to train.