stochastic gradient descent
/ stoh-KAS-tik GRAY-dee-ent dih-SENT /
Imagine you're hiking down a foggy mountain and want to reach the valley. You can't see the whole landscape, but you can feel which way the ground slopes under your feet, so you take a step downhill, feel again, and repeat. That's gradient descent. Stochastic gradient descent adds one twist: instead of carefully surveying the whole mountain before each step, you glance at just one nearby patch of ground and step based on that. It's rougher, but you take far more steps in the same time.
Concretely, plain gradient descent computes the loss over the entire training set before adjusting the weights — accurate, but painfully slow when you have millions of examples. SGD instead estimates the slope from a single example (or, in practice, a small mini-batch) and updates immediately. "Stochastic" means "randomly chosen," and that randomness makes each step noisy: it sometimes points slightly wrong. But the noise averages out over many steps, and the speed-up is enormous, which is why SGD and its descendants train essentially every large neural network today.
The noise is not just tolerated — it can help. The jiggling lets the path bounce out of shallow dips that a perfectly smooth descent might get stuck in, and it acts as a mild form of regularization. The price is that SGD rarely settles perfectly at the bottom; it tends to wander around a good region rather than land on a single point. Tuning the learning rate (step size) is what keeps that wandering productive instead of chaotic.
Training on a million images: full-batch gradient descent must process all million before taking one step — maybe a few steps per minute. SGD with a mini-batch of 128 takes a step after every 128 images — thousands of steps per minute. Each step is noisier, but vastly more steps usually wins.
Many rough steps usually beat a few perfect ones.
"Stochastic" strictly means one example per step, but in modern practice almost everyone uses mini-batches and still calls it SGD. The noisy updates are a feature, not just a compromise — but only if the learning rate is sane.