mini-batch
/ MIN-ee batch /
A mini-batch is a small handful of training examples processed together as a group before the model updates itself. It's the sensible middle ground between two extremes: looking at one example at a time (jittery and slow on modern hardware) versus looking at the entire dataset before each update (accurate but glacial). A mini-batch — say 32, 128, or 256 examples — gives you a decent estimate of which way to step, while still letting you step often.
Why batch at all? Modern processors (GPUs and the like) are built to crunch many numbers in parallel. Feeding them one example wastes most of that capacity; feeding them a batch keeps them busy. The model computes the loss across all examples in the batch, averages the gradients (the suggested adjustments), and makes a single update from that average. Larger batches give smoother, more reliable update directions; smaller batches are noisier but faster per step and often generalize a touch better.
Batch size is a genuine knob you tune, not a detail you ignore. Too small, and training is noisy and slow to converge; too large, and you may need to retune the learning rate, you eat more memory, and the extra precision can even hurt generalization. There is no universally best value — it depends on the model, the data, and the hardware you can afford, which is why people sweep over a few choices.
A dataset of 50,000 images with a mini-batch size of 100 means the model processes 100 images, makes one weight update, then grabs the next 100 — 500 updates to pass through the whole set once. Bump the batch to 500 and you get only 100 updates per pass, each based on a steadier average.
Batch size trades update frequency against update quality.
Batch size and learning rate are coupled: if you double the batch, you often need to raise the learning rate too. Changing one without the other is a classic reason training suddenly stops working.