batch size

The batch size is how many training examples the optimizer looks at before making one update to the weights. It sits at the intersection of three concerns — the quality of each gradient estimate, how well the model generalizes, and how much memory and compute you use — and trading these off well is a core part of training large vision models. Intuitively, a small batch is like asking a few people their opinion and acting on it immediately: fast and frequent, but noisy. A large batch is like polling thousands first: each decision is well-informed and steady, but you decide far less often.

The noise effect is precise. The mini-batch gradient is an average over the batch, and the variance of that average scales as roughly 1/m for batch size m, so quadrupling the batch halves the gradient noise (standard error drops as 1/√m). Larger batches therefore give smoother, more reliable gradients that let you safely use a larger learning rate. This motivates the linear scaling rule (Goyal et al., 'ImageNet in 1 Hour'): when you multiply the batch by k, multiply the learning rate by k, and use warmup to survive the larger early steps. The rule lets the same model train on hundreds of GPUs with very large batches in nearly the same number of epochs.

Generalization is the subtler axis. Keskar et al. observed a 'large-batch generalization gap': very large batches tend to converge to sharp minima that fit the training set but generalize worse, whereas the extra gradient noise of small batches nudges the model toward flatter, more robust minima. The effect is real but not absolute — careful learning-rate scaling, warmup, and longer training can close much of the gap — but it is why simply cranking the batch up for speed can quietly cost you test accuracy.

Memory is the hard constraint in vision. Activations for the forward pass must be stored for backpropagation, and that storage scales linearly with batch size and with image resolution, so high-resolution detection or segmentation models may only fit a batch of 1-4 per GPU. The standard workarounds are gradient accumulation (sum gradients over several micro-batches before one update, simulating a large batch on small hardware) and, when batch normalization is involved, being careful because tiny batches make its per-batch statistics noisy — often a reason to switch to layer, group, or synchronized batch normalization.