Standing in the dark on a landscape
Almost everything we will compute in the rest of this rung is one question wearing different clothes: given a function f that takes a vector x = (x_1, x_2, ..., x_n) and returns a single number, find the x that makes f as small as possible. That is unconstrained optimization. It sounds narrow, but maximizing is just minimizing the negative, and a staggering range of problems reduce to it: fitting a model means minimizing the mismatch between prediction and data — exactly the least-squares problem from earlier in this ladder, now with a curved error surface; training a neural network means minimizing a loss over millions of weights; designing a bridge means minimizing weight subject to it not falling down. Learn to find the bottom of a function and you have learned the computational engine under a huge slice of modern science and machine learning.
Here is the picture to hold for the whole rung. Think of f as the height of a landscape over the plane of inputs x. Minimizing f means finding the lowest point of that terrain. The cruel catch — the catch that makes optimization an algorithm rather than a formula — is that you cannot see the landscape from above. You are a hiker in thick fog at night. At your current spot you are allowed to measure exactly two things: the height f(x) right where you stand, and the steepness in every direction — which way is uphill and how steeply. From those purely local measurements, taken one point at a time, you must somehow walk to the global bottom. Every method in this rung is a different policy for which way to step and how far, using nothing but what you can feel underfoot.
The gradient: the arrow that points straight uphill
From single-variable calculus you know that f'(x) measures slope. In many dimensions the slope is no longer one number but one number per input direction, and packing them into a vector gives the gradient, written grad f(x). Its i-th entry is the partial derivative of f with respect to x_i — how fast f changes if you nudge only that one coordinate. The gradient has a beautiful geometric meaning that is worth memorizing forever: at any point, grad f points in the direction of steepest ascent, the single compass heading along which f climbs fastest, and its length is how steeply it climbs there. Turn around and you have the direction of steepest descent, -grad f: the most-downhill way to step, which is why it is the natural descent direction and the seed of the next guide.
Now the first crucial fact, the multivariable echo of 'set the derivative to zero.' If x is a genuine bottom of a valley, then the ground is flat there — flat in every direction at once. So at a minimum the gradient must vanish completely: grad f(x) = 0. This is the first-order optimality condition, and it is what almost every method secretly hunts for. Solving grad f(x) = 0 is solving a system of n equations in n unknowns, which is exactly the root-finding problem from the previous rung, only now the 'function whose root we seek' is the gradient itself. Optimization and root-finding are two faces of the same coin: minimize f, or find a zero of grad f. Keep that bridge in mind — guide 3 walks straight across it.
The Hessian: curvature, and the shape of the bowl
If the gradient is the multivariable f'(x), then the Hessian is the multivariable f''(x): the matrix of all second partial derivatives, written H or grad^2 f. Its (i, j) entry says how the i-th component of the gradient changes as you move in the j-th direction — in plain terms, how the slope bends. Where f' tells a one-dimensional curve whether it is concave up or down, the Hessian tells the landscape its curvature in every direction and every combination of directions at once. For smooth functions the Hessian is symmetric (the order of mixed partials does not matter), which is a small gift: symmetric matrices have real eigenvalues and orthogonal eigenvectors, and those eigen-directions are precisely the principal axes of the bowl — the directions in which the terrain curves most and least sharply.
This is where saddles get caught. At a stationary point, look at the eigenvalues of the Hessian. If they are all positive, the surface curves upward in every direction — you are sitting at the bottom of a genuine bowl, a true local minimum. If they are all negative, you are at a peak. If some are positive and some negative, the surface curves up some ways and down others: that is a saddle, and you should keep moving. This is the second-order optimality condition: grad f = 0 plus a positive-definite Hessian guarantees a local minimum. A symmetric matrix with all-positive eigenvalues is called positive definite, so the slogan is simply: zero gradient and a positive-definite Hessian means you have truly arrived.
The Hessian also smuggles in the central villain of this rung — conditioning — exactly the same idea from the conditioning chapter, now wearing the costume of geometry. The ratio of the largest eigenvalue to the smallest is the condition number of the Hessian, and it measures how lopsided the bowl is. When all eigenvalues are similar, the level sets are near-circular and the bowl is round; descent works beautifully. When the largest eigenvalue dwarfs the smallest, the bowl is a long, steep-walled, nearly flat-floored ravine, and a condition number near 10^8 here is just as ruinous as one when solving A x = b — it is the same number doing the same damage. That stretched ravine is the hidden cause of the zig-zagging you will meet in the very next guide.
Convexity: when local is global
Everything so far has been local — flat ground here, upward curvature here. But you wanted the lowest point in the whole landscape, and a foggy hiker who reaches a valley floor has no way to know whether a far deeper valley waits over the next ridge. In general this gap is unbridgeable: nonconvex optimization can have countless local minima and finding the global one is genuinely hard. There is, however, a blessed class of problems where the gap simply closes — convex optimization. A function is convex if its graph is a single bowl with no separate dimples: the chord between any two points on the surface lies on or above the surface, and equivalently, for a smooth f, the Hessian is positive semidefinite everywhere.
The payoff of convexity is enormous and worth stating sharply: for a convex function, every local minimum is automatically a global minimum. There are no false valleys to be trapped in, no ridge hiding something better. So a single stationary point — one solution of grad f = 0 — settles the whole problem, and a humble downhill walk is guaranteed to reach the true bottom. This is why convex problems, including the least-squares fits and linear-programming problems you will meet, are considered 'solved' in a way that training a deep network is not. Be honest about the boundary, though: most of the loss surfaces in modern machine learning are emphatically nonconvex, riddled with saddles and minima, and the methods later in this rung do not magically find the global optimum there — they find a good-enough local one, and that this works so well in practice remains partly a mystery.
Two grand strategies: line search and trust region
We have a map but not yet a way of walking. Almost every method in this rung is iterative in the same spirit as the linear solvers and root-finders before it: start at a guess x_0, and produce a sequence x_0, x_1, x_2, ... that, you hope, slides downhill toward a minimizer. The whole art is in turning each x_n into a better x_{n+1}, and there are two grand philosophies for doing it. They are worth naming now because every concrete algorithm you meet — steepest descent, Newton, BFGS, even the stochastic methods of machine learning — is a particular choice within one of these two frameworks.
The first is line search: pick a direction first, then a distance. At x_n you choose a descent direction p_n — some way that goes downhill, meaning it points into the lower half-space against the gradient — and then you ask the one-dimensional question 'how far along p_n should I go?' That single-variable subproblem is the line search, and you do not solve it exactly; you just take a step long enough to make real progress but short enough not to overshoot, a balance pinned down by conditions we will meet (the Wolfe conditions). Different methods differ only in how they pick p_n: steepest descent takes p_n = -grad f, Newton's method bends it using the Hessian, and quasi-Newton methods like BFGS approximate that bend cheaply.
The second is the trust region: pick a distance first, then a direction. Here you admit that your local information — gradient and curvature — is only trustworthy within some small radius around x_n, so you draw a circle of that radius (the trust region), build a simple model of f inside it (usually a quadratic from the Taylor expansion), and jump to that model's minimum within the circle. If the real f drops as the model promised, you trusted it well and grow the radius; if the real f disappoints, you shrink the circle and try again. The trust-region method is more robust precisely where line search is fragile — near saddles, where curvature is negative and a naive Newton step can fly off toward a maximum — because the leash of the radius never lets the step run away.
generic descent loop (line-search flavour):
x = x0
repeat:
g = grad f(x) # local slope (the gradient)
if ||g|| < tol: stop # first-order condition: grad f ~ 0
p = choose_direction(g, H) # -g, or a Newton/quasi-Newton bend
a = line_search(f, x, p) # how far along p (a > 0)
x = x + a * p # take the step
the methods of this rung differ ONLY in choose_direction:
steepest descent : p = -g
Newton : solve H p = -g (uses curvature)
quasi-Newton : p = -B*g, B approximates the inverse HessianWhat is local, what is honest, and where we go next
Before marching on, hold onto the assumptions, because forgetting them is how optimization goes wrong in practice. Everything here rests on f being smooth enough to have a gradient and a Hessian; kinks and jumps break the whole picture and need different tools. The gradient and Hessian are themselves usually computed approximately — by finite differences, which carry the same step-size dilemma and round-off floor as numerical derivatives, or far better by automatic differentiation, the workhorse behind every deep-learning framework. And the iteration lives in floating-point: 'grad f = 0' is never reached exactly, so in practice you stop when the gradient is merely small, and on an ill-conditioned bowl a small gradient can still sit a long way from the true minimum. Every minimizer you compute is an approximation, with a quality set, as always in this subject, by conditioning times stability.
That is the landscape and the toolkit. The gradient gives the first-order condition grad f = 0; the Hessian's eigenvalues give the second-order test that separates a true minimum from a saddle and quietly set the conditioning of the bowl; convexity is the gift that makes local global; and line search versus trust region are the two ways to actually walk. The rest of the rung is this skeleton fleshed out. Guide 2 takes the most obvious direction, p = -grad f, and discovers it crawls in a maddening zig-zag down ill-conditioned ravines. Guide 3 uses the Hessian to bend the step into Newton's method and its cheaper quasi-Newton cousins, fixing the zig-zag. Guide 4 throws away exactness for speed with stochastic gradient descent, the engine of machine learning. And guide 5 puts a fence around the search — constraints — and meets Lagrange multipliers and the KKT conditions. You now have the map; the next four guides are about learning to walk it well.