the Nelder-Mead method
/ NEL-der MEED /
What if you want to minimize a function but have no formula for its slope — only a black box that, given an input, returns a number (maybe from an experiment, a simulation, or messy code)? You cannot compute a gradient, so gradient descent and Newton's method are off the table. The Nelder-Mead method solves exactly this: it minimizes using only function VALUES, by crawling a flexible shape across the landscape like an amoeba feeling its way downhill.
The 'shape' is a simplex — for n variables, a figure with n+1 vertices (a triangle in 2-D, a tetrahedron in 3-D). At each step the method evaluates f at all vertices, finds the WORST (highest) one, and tries to improve it by moving it through the opposite face: it REFLECTS the worst vertex to the other side of the simplex; if that point is excellent it EXPANDS further; if reflection is poor it CONTRACTS inward; and if nothing helps it SHRINKS the whole simplex toward the best vertex. Through these four moves — reflect, expand, contract, shrink — the simplex tumbles, stretches, and squeezes its way toward a minimum, adapting its size and shape to the terrain, never needing a single derivative.
Nelder-Mead is popular and widely used precisely because it is derivative-free, simple, and robust to mild noise — ideal for tuning a handful of parameters of a black-box objective, calibrating a simulation, or any low-dimensional problem where gradients are unavailable or unreliable. The honest limitations are important. It has essentially NO convergence guarantee: there are documented examples where it converges to a non-stationary point (it can stall away from any minimum), and it can stagnate as the simplex collapses to a lower-dimensional sliver. It scales POORLY — it is practical only for small n (say up to 10 or 20 variables), as the simplex becomes unwieldy in high dimensions. When derivatives ARE available, gradient-based methods are far faster and better-grounded. Nelder-Mead is a convenient tool of last resort for derivative-free, low-dimensional problems, not a general-purpose optimizer.
To minimize a noisy 2-D experimental yield f(temperature, pressure) with no derivative available, start with a triangle of three (T, P) trials. Evaluate all three, reflect the worst-yielding corner across the other two to a new trial, expand if it is great, contract if it is poor. The triangle creeps and reshapes toward the best operating point — pure function values, no gradient.
A simplex reflects, expands, contracts, and shrinks — no derivatives needed.
Nelder-Mead has no general convergence guarantee — it can stall at a non-stationary point and degrades badly above ~10-20 variables. Use it only when derivatives are truly unavailable and the dimension is small; if you can compute or auto-differentiate a gradient, a gradient-based method is faster and far better grounded.