stochastic gradient descent

Imagine you are blindfolded on a hilly landscape and want to reach the lowest valley. The honest way is to feel the slope under your whole body and step downhill; in machine learning that means computing the gradient of the loss over the entire training set before each step. That is accurate but ruinously slow when the set has millions of images. Stochastic gradient descent (SGD) makes a bargain: instead of feeling the full slope, it samples a small random handful of examples (a mini-batch), measures the slope just there, and steps immediately. Each step is a noisy guess at the true downhill direction, but you take thousands of steps in the time the careful method takes one, and the guesses average out toward the right way down.

Precisely, let the parameters (all the weights of the network) be a vector θ, and let the training loss be the average of a per-example loss ℓ_i over the dataset of N examples: L(θ) = (1/N) Σ_i ℓ_i(θ). Full-batch gradient descent updates θ ← θ − η ∇L(θ), where ∇L is the gradient (the vector of partial derivatives pointing in the direction of steepest increase, so we subtract it to go down) and η (eta) is the learning rate, a small positive step size. SGD instead draws a random mini-batch B of m examples and uses ĝ = (1/m) Σ_{i∈B} ∇ℓ_i(θ) as a cheap estimate of ∇L, updating θ ← θ − η ĝ. Because the batch is sampled uniformly at random, ĝ is an unbiased estimator of the true gradient: on average it equals ∇L, even though any single batch is off by some random amount.

That random amount is the crux. The variance (the expected squared error) of the mini-batch gradient scales roughly as 1/m, so small batches are noisier and large batches smoother. This noise is not purely a nuisance: it acts as an implicit regularizer, jittering the parameters so they slide out of sharp, brittle minima and tend to settle in flatter basins that generalize better to new data. It also helps the optimizer escape saddle points (flat spots where the full gradient is near zero) that would stall a deterministic method.

The learning rate η governs how big each step is and is the single most consequential knob. Too large and the steps overshoot the valley and the loss oscillates or diverges to NaN; too small and training crawls. Classical stochastic-approximation theory (Robbins and Monro, 1951) shows convergence on convex problems requires the learning rate to shrink over time so that Σ η_t = ∞ but Σ η_t² < ∞. In modern deep learning we rarely satisfy those conditions literally; instead we use a fixed or scheduled learning rate and accept that SGD wanders in a noise ball around a good solution rather than converging to a single point.