the Wolfe conditions
/ WOOLF /
When you take a step downhill, two things can go wrong. Take too BIG a step and you overshoot, climbing the far side of the valley — you wanted to descend but rose. Take too SMALL a step and you barely move, making no real progress and dooming the method to inch along forever. The Wolfe conditions are a pair of tests that a line-search step length must pass to be 'just right': big enough to make real progress, small enough to actually go down.
Step the iterate from x_k by alpha along direction p_k, and let phi(alpha) = f(x_k + alpha p_k). The first Wolfe condition is the ARMIJO (sufficient-decrease) condition: f(x_k + alpha p_k) <= f(x_k) + c_1 * alpha * grad f(x_k)^T p_k, with a small c_1 (typically 1e-4). It says the actual drop must be at least a fraction c_1 of the drop the initial slope predicts — this forbids steps that are too long (overshooting). The second is the CURVATURE condition: grad f(x_k + alpha p_k)^T p_k >= c_2 * grad f(x_k)^T p_k, with c_1 < c_2 < 1 (typically c_2 = 0.9). It demands the slope at the new point be less steep (closer to flat) than at the start, which forbids steps that are too short. Together (Armijo plus curvature) they are the Wolfe conditions; replacing the curvature test with |grad f(...)^T p_k| <= c_2 |grad f(x_k)^T p_k| gives the stronger STRONG Wolfe conditions.
These conditions matter because they are the contract that makes line-search methods converge, and they are essential to quasi-Newton methods: BFGS and L-BFGS rely on the curvature condition to keep their approximate Hessian positive definite (the update would break otherwise). It is a theorem (Wolfe's) that for a smooth function bounded below, step lengths satisfying these conditions always exist along any descent direction, so a line search can always succeed. The honest nuance: c_1 must be small and strictly less than c_2, or the two requirements conflict and no step satisfies both.
Minimizing phi(alpha) = f(x_k + alpha p_k) with c_1 = 1e-4, c_2 = 0.9: a tiny step alpha = 0.001 passes Armijo (it does go down) but fails curvature — the slope is still steeply negative, signalling 'too short, push further'. A huge alpha that overshoots fails Armijo — it rose more than allowed. A moderate alpha near the valley passes both: real progress, slope nearly flat.
Armijo bans overshooting; curvature bans dawdling — together, just right.
The Armijo condition alone allows absurdly tiny steps; the curvature condition is what rules those out and keeps quasi-Newton's Hessian update positive definite. Use c_1 small (around 1e-4) and c_2 larger (0.9 for Newton-type, ~0.1 for nonlinear CG), with c_1 < c_2 — otherwise no step can satisfy both.