local minima and saddle points

When training descends the loss landscape looking for the lowest point, it can get stuck or stalled at places that aren't the true bottom. A local minimum is a dip that's lowest in its immediate neighborhood but not lowest overall — like a small pond partway down a mountain. Stand in it, look around, and every direction goes up, even though a deeper valley exists somewhere else. The global minimum is the actual lowest point of all.

A saddle point is subtler and, it turns out, far more common in high dimensions. It's a spot that goes up in some directions and down in others — shaped like a horse's saddle, or a mountain pass. The slope there is flat (zero gradient), so naive descent slows to a near-stop, fooled into thinking it has arrived, even though escape routes downhill exist if you look in the right direction. Long plateaus in a learning curve are often saddle points being slowly traversed.

Here's the reassuring, somewhat surprising reality for deep learning: in the very high-dimensional landscapes of large neural networks, bad local minima are rarer than intuition from low-dimensional pictures suggests, and most troublesome flat spots are saddle points, not traps. For a point to be a true local minimum, it must curve upward in every one of millions of directions at once — statistically unlikely. This is part of why simple gradient descent works as well as it does, and why momentum and noisy SGD updates, which help slide off saddle points, are so useful.