unconstrained optimization

Picture a landscape of hills and valleys whose height at the point (x, y) is given by a function f. Unconstrained optimization is the problem of finding the highest peaks and lowest basins of that surface when you are free to stand anywhere — there is no fence, no curve you must walk along, no budget you must respect. The variables can roam over the whole space (or some open region), and you simply ask where f is largest or smallest.

The key tool is the gradient, the vector of partial derivatives (df/dx, df/dy, ...). At a smooth interior peak or valley the surface is locally flat, so every partial derivative is zero: the gradient vanishes. Solving grad f = 0 finds the candidate points, called stationary or critical points. To then decide whether a candidate is a maximum, a minimum, or neither, you look at the second derivatives gathered in the Hessian matrix and check its definiteness — the multivariable version of the single-variable second-derivative test.

This is the optimization that underlies curve fitting, maximum-likelihood estimation in statistics, energy minimization in physics and chemistry, and the training of machine-learning models. The honest caveat: grad f = 0 only locates interior stationary points. A true global optimum may instead sit on the boundary of the region, or escape to infinity, or fail to exist at all if f is unbounded. Setting the gradient to zero is necessary for an interior optimum, not sufficient to guarantee one exists.