edge of stability

Classical optimization theory gives a strict speed limit: for gradient descent to decrease the loss monotonically, the step size must keep the curvature below two divided by the learning rate. Train a real network with full-batch gradient descent, though, and it does the opposite of obeying. The curvature climbs until it reaches that very threshold, then the optimizer hovers right at the edge — loss bouncing up and down step to step yet drifting steadily downward over the long run.

Sharpness here means the largest eigenvalue of the loss Hessian. Two effects combine: progressive sharpening drives that top eigenvalue upward during training until it hits two over the learning rate, after which the dynamics enter the edge-of-stability regime where sharpness stalls at roughly that value and the loss is non-monotone but still decreases. This flatly contradicts the classical descent-lemma picture, in which exceeding the threshold should cause divergence, not productive training.

The edge of stability reframes how large-step gradient descent actually behaves on neural networks and connects step size directly to the curvature the optimizer is willing to tolerate — a reminder that the smooth-optimization intuitions imported from convex theory only partly survive contact with deep learning.