BFGS
/ B-F-G-S, spelled out /
BFGS is the single most popular quasi-Newton method — the default many software libraries reach for to minimize a smooth function. Its name stacks the four people who discovered the same update independently in 1970: Broyden, Fletcher, Goldfarb, and Shanno. The idea, like all quasi-Newton methods, is to enjoy Newton's curvature-driven speed while building the curvature from gradients alone, never computing or storing the true Hessian.
BFGS maintains an approximation to the inverse Hessian, call it M_k, so the step is simply a matrix-vector product p = -M_k grad f(x_k) (no linear solve needed). After moving with step s_k = x_{k+1} - x_k and observing the gradient change y_k = grad f(x_{k+1}) - grad f(x_k), it applies the BFGS update — a rank-two correction to M_k that (a) satisfies the secant equation H s_k = y_k and (b) provably KEEPS M_k symmetric positive definite as long as the curvature condition y_k^T s_k > 0 holds (which a Wolfe line search guarantees). Positive definiteness matters because it ensures p is always a genuine descent direction. The full BFGS stores the whole n-by-n matrix; its famous variant LIMITED-MEMORY BFGS (L-BFGS) stores only the last m pairs (s_k, y_k) — typically m = 5 to 20 — and reconstructs the matrix-vector product implicitly, dropping the memory from O(n^2) to O(m n) so it scales to millions of variables.
BFGS and L-BFGS are everywhere: training logistic regression and conditional random fields, fitting scientific models, and any smooth medium-to-large minimization where gradients are available. They achieve superlinear convergence using only first derivatives, are robust, and need little tuning. The honest scope: they assume a reasonably smooth objective and reliable gradients; they are NOT the right tool for the noisy, mini-batch, stochastic objectives of large-scale deep-learning training (where SGD-family methods like Adam dominate, because the secant-update curvature estimate is corrupted by gradient noise). For deterministic, full-gradient smooth problems, though, L-BFGS is often the method to beat.
To fit a logistic-regression model with 50,000 weights, L-BFGS keeps just the last 10 gradient-difference pairs — about 1 MB — instead of a 50,000-by-50,000 inverse-Hessian (20 GB, impossible). It converges in tens of iterations, each a few cheap inner products, far faster than plain gradient descent and far lighter than full Newton.
L-BFGS stores a few vectors, not a matrix — Newton-ish on a memory budget.
BFGS shines on smooth, deterministic, full-gradient problems but is the WRONG default for mini-batch deep learning: stochastic gradient noise corrupts the secant curvature estimate, so the update misbehaves. For noisy objectives, SGD with momentum or Adam is the standard, not (L-)BFGS.