nonlinear least squares
Linear least squares works because the model depends linearly on its unknowns, giving a one-shot closed-form answer. But many real models are not so kind — fitting an exponential decay y = a*exp(-b t), a sigmoid, or a sum of Gaussians means the parameters enter nonlinearly. Nonlinear least squares still minimizes the sum of squared residuals, but now there is no formula; you must search for the best parameters iteratively, improving a guess step by step until the residuals stop shrinking.
The objective is to minimize sum_i r_i(p)^2 where each residual r_i(p) = y_i - f(x_i; p) depends nonlinearly on the parameter vector p. The dominant strategy linearizes the model around the current guess: form the Jacobian matrix J (the derivatives of the residuals with respect to the parameters), and approximate the curved problem near p as a LINEAR least-squares problem in the step. That is the Gauss-Newton method: at each iteration solve the linear least-squares system J delta = -r for a correction delta, update p -> p + delta, recompute, and repeat. Near a good fit with small residuals it converges fast, almost like Newton's method, because the linear approximation is excellent there.
Two honest cautions distinguish this from the linear case. First, it is genuinely an optimization problem, so it can stall at a local minimum, depend heavily on the starting guess, and is not guaranteed to find the global best fit — a previewing taste of numerical optimization. Second, plain Gauss-Newton can overshoot or diverge when the residuals are large or the model is badly nonlinear; the standard robust fix is the Levenberg-Marquardt method, which blends Gauss-Newton with a cautious gradient-descent step. Despite these caveats, nonlinear least squares is the everyday tool of scientific curve fitting, parameter estimation, and model calibration.
Fitting y = a*exp(-b t) to decay data: with a guess (a, b), compute residuals r_i = y_i - a*exp(-b t_i) and the Jacobian of partials with respect to a and b, solve the linear least-squares step J delta = -r, update (a, b), and iterate until the fit settles. The parameter b enters nonlinearly, which is why no closed form exists.
Linearize, take a least-squares step, repeat — Gauss-Newton turns a curved fit into a sequence of linear ones.
Unlike linear least squares, the nonlinear version is a true optimization problem: it can converge to a local rather than global minimum and depends on the starting guess. A poor initial point can give a confidently wrong fit.