The most natural idea: walk downhill
In the previous guide you learned the shape of the landscape: a function f to be minimized, its gradient grad f(x) that points in the direction of steepest increase, and the Hessian that bends the surface into bowls, ridges, and saddle points. The first-order optimality condition told you a minimum can only sit where grad f(x) = 0. Now we want an algorithm that actually finds such a point. The most natural idea in all of optimization is almost embarrassingly simple: if the gradient points uphill, then its negative points downhill, so just walk that way.
That single sentence is gradient descent (also called steepest descent). From your current point x_n you compute the gradient, step in the opposite direction by some amount, and repeat. Written out, the update is x_{n+1} = x_n - alpha * grad f(x_n), where the positive number alpha is the step size (in machine learning it is called the learning rate). The negative gradient is the canonical example of a descent direction: a direction d along which f decreases, at least for a small enough step, because the directional derivative grad f(x)·d is negative there.
gradient descent, the whole algorithm:
x = x0 # starting guess
repeat:
g = grad_f(x) # the gradient at the current point
if ||g|| < tol: # gradient ~ 0 => near a stationary point, stop
break
x = x - alpha * g # one step downhill
the entire method is that one line x = x - alpha*g , repeated.
the only real decisions are: the starting point x0, and how big alpha is.How big a step? The Goldilocks problem
The direction is settled — straight downhill — but the distance alpha is a genuine choice, and it is where gradient descent earns its bruises. The gradient tells you which way is down right here, but it says nothing about how far the slope stays steep. Take too small an alpha and each step barely moves; you inch toward the minimum and burn thousands of iterations. Take too large an alpha and you can overshoot the valley floor and land higher up the far wall, sometimes bouncing to ever-larger heights until the numbers blow up to infinity. There is a 'just right' middle, and finding it is the whole practical game.
There are two honest ways to handle the step. The simplest is to fix alpha at some small constant and hope — common in machine learning, where it is tuned by trial. The principled way is a line search: at each iteration, look along the chosen direction as a one-dimensional problem and pick a step that genuinely reduces f by enough. You do not need the exact best step (that would be its own little minimization); you need a 'good enough' one. The Wolfe conditions make 'good enough' precise: one inequality demands that f actually decreases in proportion to how steep the slope was (so you do not creep), and another demands that the slope has flattened enough (so you do not stop too short). A step meeting both is accepted, and the method is guaranteed to keep making progress.
The zig-zag: when steepest is not shortest
Here is the picture to burn into memory. Imagine the cost surface is a long, narrow valley — a taco shell, steep across its width but nearly flat along its length. The minimum sits far down at the centerline. Now stand on one steep wall and ask the gradient where to go. It points almost straight across the valley toward the opposite wall, only very slightly down the length, because the surface is steepest across, not along. So you step across, overshoot the centerline, land on the opposite wall, and the gradient there points right back the way you came. The result is a slow zig-zag, ricocheting from wall to wall while crawling only a sliver closer to the true minimum each bounce.
This is not bad luck or a coding bug; it is built into the method. The negative gradient is the steepest direction only in an infinitesimal sense, valid for one infinitesimal instant. The moment you take a finite step it can be badly wrong, and in a narrow valley it is wrong by nearly ninety degrees. What governs how bad it gets is the shape of the bowl, captured by the Hessian's eigenvalues: the ratio of its largest to its smallest eigenvalue is the condition number of the problem. A round bowl has condition number near 1 and gradient descent walks almost straight in; a long thin valley has a large condition number and gradient descent zig-zags. This is the very same conditioning idea from the linear-algebra rung — a problem with a condition number of 10^4 stretches the bowl 10^4 times longer than it is wide, and the crawl is correspondingly brutal.
Make it concrete with the cleanest possible example: f(x,y) = (1/2)(x^2 + 100 y^2), whose minimum is at the origin and whose condition number is exactly 100/1 = 100. The gradient is (x, 100 y), so one step of gradient descent shrinks x by the factor (1 - alpha) and y by the factor (1 - 100 alpha). To keep the steep y-direction from overshooting and diverging you must keep alpha below 2/100 = 0.02 — but that very same small alpha shrinks the flat x-direction by only (1 - 0.02) = 0.98 per step, removing a mere 2% of the x-error each iteration. The stiff direction caps the step size; the flat direction then pays for it, needing hundreds of steps to converge. That single tug-of-war, between the largest and smallest eigenvalue, is the whole zig-zag in miniature.
How slow is slow? The honest rate
It pays to be precise about the cost, because 'slow' has a formula. On a nicely curved (convex) problem gradient descent converges linearly: the distance to the minimum is multiplied by a fixed factor each step, a factor of roughly (cond - 1) / (cond + 1), where cond is the condition number of the Hessian. When cond is near 1 that factor is near 0 and you converge in a handful of steps. When cond = 100 the factor is about 0.98, so each step removes only 2% of the error and you need on the order of a hundred steps to gain one digit of accuracy. When cond = 10^4 you need tens of thousands. This is what 'crawls on ill-conditioned problems' means quantitatively, and it is the headline weakness of the method.
Note the contrast in vocabulary, because it matters. 'Linear convergence' here sounds fast but is actually the slow regime — the error shrinks by a constant factor each step, so the number of correct digits grows linearly with the iteration count. Compare Newton's method in the next guide, which uses the Hessian to converge quadratically near the minimum: the number of correct digits roughly doubles each step. Gradient descent's rate of convergence is the price you pay for using only first derivatives and never building the Hessian.
Two honest caveats keep this from being a hit piece. First, the whole clean story — a single minimum, a guaranteed downhill march — relies on convexity: a bowl-shaped surface with no false bottoms. On a non-convex landscape (the norm in deep learning) gradient descent can stall at a saddle point or settle into a local minimum that is not the global one, and the tidy rate above no longer holds. Second, every number it produces is an approximation reached by stopping early; you halt when the gradient is small enough or the steps stop helping, and in finite-precision arithmetic a 'zero' gradient is really a gradient lost in round-off, so there is a floor below which more iterations buy nothing.
Fixing the crawl, and what comes next
Once you see that the zig-zag comes from ill-conditioning, the cures organize themselves around a single goal: make the bowl rounder, or stop fighting its shape. The deepest cure is to use second-derivative information — the Hessian tells you the bowl is stretched and rescales the step accordingly, turning the taco shell back into a circle. That is Newton's method and its cheaper cousins, the subject of the next guide. They trade the cost of forming and solving with the Hessian for a dramatically better direction and quadratic convergence near the answer.
But there is a cheaper, beautifully simple fix that stays first-order: momentum. Instead of stepping purely along today's gradient, you keep a running average of recent gradients and step along that. In the narrow valley the across-the-valley components keep flipping sign and cancel out, while the small but consistent along-the-valley component accumulates — like a marble that ignores the side-to-side jitter and rolls steadily down the trough. Nesterov's accelerated gradient is a refined version of this idea, and on convex problems it provably improves the rate from depending on cond to depending on sqrt(cond), a large win when cond is huge. Momentum is what makes plain gradient descent practical at the scale of modern neural networks.