momentum
/ moh-MEN-tum /
Momentum borrows an idea from physics: a heavy ball rolling downhill doesn't stop and rethink at every pebble — it builds up speed and keeps rolling in the direction it's been going. In training, momentum makes the model remember the direction of its recent steps and keep moving that way, instead of reacting only to the slope right under its feet. The result is a smoother, faster descent.
Mechanically, plain gradient descent steps purely in the current downhill direction. Momentum instead keeps a running average of past gradients — a "velocity" — and steps in that accumulated direction. A typical momentum value of 0.9 means each update is mostly the previous velocity plus a little of the new gradient. When successive gradients agree, they pile up and the model accelerates down the long, consistent slopes; when they disagree (the noisy, zig-zagging directions), they partly cancel, damping the wobble.
This is genuinely useful in two common situations: long gentle valleys where plain descent inches along, and narrow ravines where it ping-pongs from wall to wall. Momentum powers through the first and smooths out the second. The cost is that a heavy ball can overshoot — roll past the bottom before turning around — so momentum is paired with a sensible learning rate. It's the foundation underneath the popular Adam optimizer, which adds momentum to per-parameter step sizing.
Picture a narrow ravine sloping gently downhill. Plain gradient descent bounces left-right between the steep walls, making slow forward progress. Add momentum and the side-to-side bounces cancel out while the gentle forward direction accumulates — the model glides down the ravine instead of rattling along it.
Momentum cancels the zig-zag and keeps the useful forward motion.
Momentum doesn't replace the learning rate — it works alongside it. Too much momentum plus too large a learning rate is a recipe for overshooting and oscillation, so the two are tuned together.