Numerical Optimization

the first-order optimality condition

Stand at the bottom of a valley and feel the ground under your feet: it is level. Stand at the top of a hill: also level. Anywhere the ground tilts, you could step downhill and go lower — so a lowest point can never be on a slope. The first-order optimality condition turns this everyday fact into a test: at any minimum (or maximum) of a smooth function, the ground must be flat — every directional slope is zero.

Precisely: if x* is a local minimum of a differentiable function f and x* is not on a boundary, then the gradient vanishes, grad f(x*) = 0. The gradient is the vector of partial derivatives (df/dx_1, ..., df/dx_n); it points in the direction of steepest increase, and its length is the steepest slope. Setting it to the zero vector says no direction goes downhill — locally the surface is level. A point where grad f = 0 is called a STATIONARY point or critical point, and solving grad f(x) = 0 is how calculus locates candidate minima. This is a system of n equations in n unknowns, often nonlinear, which is itself solved numerically (for example by Newton's method).

The crucial honesty: this condition is NECESSARY but not SUFFICIENT. Flat ground marks a minimum, a maximum, OR a saddle point — all three have zero gradient. So grad f(x*) = 0 only nominates x* as a candidate; you need the second-order condition (curvature) to tell which it is. Descent algorithms in practice never solve grad f = 0 exactly; they stop when the gradient's length falls below a small tolerance, ||grad f|| < epsilon, accepting a point that is 'flat enough'.

For f(x) = x^3 - 3x, the gradient is f'(x) = 3x^2 - 3, zero at x = 1 and x = -1. Both are stationary points, yet x = 1 is a local minimum and x = -1 a local maximum. The first-order test flags both equally; only curvature (f''(x) = 6x) separates them.

Zero gradient nominates a candidate but cannot say minimum, maximum, or saddle.

Zero gradient is necessary only for INTERIOR minima of smooth functions. At a constraint boundary or a kink (a non-differentiable point, like |x| at 0), the minimum can sit where the gradient is nonzero or undefined — that is why constrained problems need the KKT conditions instead.

Also called
stationary pointgradient equals zerovanishing gradient梯度為零駐點條件