the Levenberg-Marquardt method
/ LAY-ven-berg MAR-kwart /
The Levenberg-Marquardt method is the trusty default for nonlinear least squares — the algorithm that quietly powers most curve-fitting routines. It solves the same problem as Gauss-Newton but adds a safety valve. Gauss-Newton is fast but reckless: when the current guess is far from the answer, its big linearized steps can overshoot and diverge. Plain gradient descent is the opposite: always moves downhill safely, but creeps painfully slowly. Levenberg-Marquardt blends the two, leaning on whichever is appropriate moment by moment.
Mechanically, Gauss-Newton solves (J^T J) delta = -J^T r for each step. Levenberg-Marquardt instead solves the DAMPED system (J^T J + mu D) delta = -J^T r, where mu >= 0 is a damping parameter and D is a positive diagonal (often diag(J^T J)). When mu is small, the term J^T J dominates and you get a fast Gauss-Newton step; when mu is large, the damping dominates and you get a short, safe step in the gradient-descent direction. The algorithm adapts mu on the fly: after a step that reduces the residual, it shrinks mu to be bolder; after a step that fails, it grows mu to be more cautious and retries. This is exactly a trust-region strategy — mu implicitly limits how far each step may roam.
The payoff is robustness: Levenberg-Marquardt converges from poorer starting guesses than Gauss-Newton while keeping near-quadratic speed close to the solution, which is why it is the workhorse of scientific parameter estimation, computer-vision calibration, and machine-learning model fitting on small problems. The honest caveats carry over from nonlinear least squares: it finds a LOCAL minimum, not necessarily the global one, and the damping notwithstanding, a bad initial guess or a poorly identifiable model can still defeat it. It is robust, not magic.
When fitting a sum of Gaussians from a far-off guess, early iterations have a large mu, so Levenberg-Marquardt takes small cautious downhill steps and avoids the wild overshoot that plain Gauss-Newton would suffer. As the fit improves, mu shrinks automatically and the steps become fast Gauss-Newton steps, snapping onto the local minimum.
Damping interpolates between safe gradient descent (large mu) and fast Gauss-Newton (small mu).
Levenberg-Marquardt is robust but still local: it converges to a minimum near the starting point, not necessarily the global best fit. For multimodal problems you still need good initial guesses or a global search on top.