steepest descent
Lost on a foggy hillside and wanting to reach the valley fast, the most natural move is to face whichever way the slope drops most steeply and step that way — then look again and repeat. Steepest descent (the deterministic gradient descent on a full function) is exactly this greedy rule: at every point, walk straight in the direction of fastest decrease, which is the opposite of the gradient.
The gradient grad f(x) points in the direction of steepest INCREASE, so its negative -grad f(x) points in the direction of steepest decrease. The iteration is x_{k+1} = x_k - alpha_k grad f(x_k), where alpha_k > 0 is a step size (learning rate) set by a line search or fixed by hand. Each step uses only first-derivative (gradient) information — no Hessian, no matrix to store or invert — so a step is cheap, O(n) work for n variables, which is why it scales to enormous problems. On a perfectly round bowl it heads straight for the center.
Its famous weakness is ZIG-ZAGGING on ill-conditioned problems. When the valley is a long, narrow ravine (the Hessian's eigenvalues differ greatly, a large condition number kappa), the steepest-descent direction points mostly across the ravine, not along it; successive steps bounce off the steep walls and crawl down the gentle floor in a slow zig-zag. The number of iterations to converge grows roughly like the condition number kappa, so a stretched-out problem (kappa = 10^4) can take thousands of steps. This is precisely why people precondition, or switch to conjugate-gradient, Newton, or quasi-Newton methods, which account for curvature and cut straight down the ravine. Also note exact line searches make consecutive steepest-descent steps orthogonal, which is geometrically why it bounces from wall to wall.
Minimize f(x, y) = x^2 + 100 y^2 (a stretched bowl, condition number 100). The gradient is (2x, 200y); at (1, 1) it is (2, 200), pointing almost straight along y. Steepest descent shoots mostly in -y, overshoots, then corrects, then overshoots again — zig-zagging down the narrow trough. A round bowl x^2 + y^2 instead reaches the center in one step.
On a round bowl it dives straight in; on a stretched one it zig-zags.
Steepest descent is locally greedy, not globally efficient: it converges only LINEARLY, and slowly when the problem is ill-conditioned (iterations scale with the condition number). The fix is curvature — preconditioning, conjugate gradient, or (quasi-)Newton — not a smaller step size, which only makes it crawl more.