gradient

Stand again on that hill where height depends on your east-west and north-south position. At your feet there is one direction in which the ground climbs most steeply — the way water would flow if it ran backward, uphill. The gradient is an arrow pointing exactly that way: the direction of fastest increase. Its length tells you how steep that steepest climb is. Turn ninety degrees from it and you walk along the level, neither rising nor falling.

Mathematically, the gradient of a function f(x, y, ...) bundles all its partial derivatives into one vector: grad f = (∂f/∂x, ∂f/∂y, ...), often written with the nabla symbol as ∇f. Each component is the slope along one axis; together they assemble into the single direction of steepest ascent, and the gradient is always perpendicular to the level curves or surfaces where f stays constant.

This idea is the engine of modern optimization and machine learning. To find a minimum of a cost function — say, to train a model — you repeatedly step in the direction opposite the gradient, downhill, a procedure called gradient descent. One honest caveat: the gradient only describes the local lay of the land. Following it downhill can settle you in a nearby low valley that is not the lowest point overall.