momentum

Plain SGD steps in the direction of the current mini-batch gradient and then forgets it. Momentum gives the optimizer memory and inertia, exactly like a heavy ball rolling downhill: the ball does not stop and re-decide its direction at every instant; it keeps a velocity built up from where it has been. When gradients consistently point the same way, the velocity grows and the ball accelerates down long gentle slopes. When the gradient flips back and forth across a narrow ravine (a common shape near minima), the opposing components cancel in the running average while the consistent down-the-valley component survives, so momentum damps the wasteful side-to-side oscillation and speeds progress along the floor.

Concretely we maintain a velocity vector v, initialized to zero, and on each step blend the old velocity with the new gradient g before moving: v ← μ v + g, then θ ← θ − η v. Here μ (mu) is the momentum coefficient, a number in [0,1) that says how much of the past to keep, typically 0.9; η is the learning rate and θ the parameters. With μ = 0 this reduces to ordinary SGD. The velocity is an exponential moving average of past gradients, so a gradient's influence decays as μ, μ², μ³, … over later steps. In the steady state where the gradient is roughly constant, the velocity converges to g/(1−μ), so momentum effectively amplifies the step size by a factor 1/(1−μ) — about 10× at μ = 0.9. This is why you often must lower the learning rate when you add momentum.

Nesterov accelerated gradient (NAG) is a sharper variant that evaluates the gradient at a look-ahead position — where the velocity is about to carry the parameters — rather than at the current point. Intuitively the ball checks the slope where it is heading, not where it is standing, letting it brake before overshooting. For smooth convex problems Nesterov's method achieves a provably better convergence rate than plain gradient descent, and in practice it often gives a small but reliable edge over classical momentum.

In computer vision, SGD with momentum (μ = 0.9) plus weight decay and a learning-rate schedule is the workhorse that trained the landmark CNNs — AlexNet, VGG, and the ResNet family — and it still frequently matches or beats adaptive optimizers like Adam on convolutional image classifiers, often generalizing slightly better. Adaptive methods tend to dominate for transformers, but momentum remains a strong, simple default.