sharpness-aware minimization (SAM)

Two networks can reach exactly the same training loss yet generalize very differently. A long-standing intuition is that sharp minima, where the loss rises steeply if you nudge the weights, generalize worse than flat minima, where the loss stays low across a whole neighborhood. Sharpness-aware minimization turns that intuition into an explicit objective: do not just find weights with low loss, find weights whose entire surrounding region has low loss.

Formally SAM minimizes, over theta, the worst-case loss in a ball of radius rho around theta, a min-max problem. The inner maximization is approximated by a single gradient ascent step, giving a perturbation epsilon-hat equal to rho times the normalized gradient; SAM then evaluates the gradient at the perturbed point theta plus epsilon-hat and uses it to update theta. Each step therefore costs two forward-backward passes instead of one.

Empirically SAM improves generalization on vision and language benchmarks and adds robustness to label noise, which is consistent with the flat-minima story. The downsides are the doubled gradient cost, partly addressed by efficient and adaptive variants, and an extra hyperparameter rho that sets the neighborhood radius and genuinely needs tuning for the effect to show up.