steepest descent
Imagine you are standing on a foggy hillside and want to reach the bottom as fast as possible, but you can only feel the slope under your feet. The most sensible move is to step in the direction that goes down most steeply right now. Steepest descent formalizes exactly this instinct into an iterative method for minimizing a function of several variables.
The gradient grad f points in the direction of fastest increase of f, so its negative, minus grad f, points in the direction of fastest decrease — the steepest downhill direction at your current point. The method moves a step along that direction, recomputes the gradient at the new point, and repeats, generating a sequence x_new = x_old - t times grad f, where the step length t is chosen to make good progress (an exact line search picks the t that minimizes f along that ray). Because each step heads straight downhill, the value of f decreases at every iteration until the gradient is nearly zero, signalling a stationary point.
Steepest descent is the conceptual ancestor of nearly all first-order optimization. Its honest weaknesses are well known. On long narrow valleys it zigzags badly: because each step is orthogonal to the last, it crosses the valley floor back and forth instead of running down its length, and convergence crawls. And it only finds a local minimum — wherever the downhill path happens to lead. These limitations motivate smarter variants (conjugate gradients, momentum, Newton-type methods), but the core idea, follow the negative gradient, remains the heartbeat of how almost all large models are trained.
For f(x, y) = x^2 + 10 y^2, the gradient at (1, 1) is (2, 20). The steepest-descent step heads along (-2, -20) — overwhelmingly toward smaller y. On this stretched bowl the iterates zigzag, illustrating why an ill-conditioned problem makes plain steepest descent slow.
Steepest descent zigzags on elongated valleys — successive steps are orthogonal, so progress along the valley is slow.
The steepest direction is steepest only locally and only in the chosen coordinates — it is not the direct line to the minimum. That is why pure steepest descent, though always downhill, can be inefficient.