Tikhonov regularization

When a least-squares problem is ill-conditioned, the honest best fit can be wildly unstable: a hair of noise in the data sends the coefficients to extremes, and the inverse barely exists. Tikhonov regularization tames this by gently discouraging large coefficients. Instead of just minimizing the misfit, you also penalize the size of the solution, trading a tiny bit of fit for a great deal of stability. In statistics the same idea is called ridge regression — and the picture is: do not chase the data so hard that you amplify its noise.

Concretely you minimize ||A x - b||_2^2 + lambda^2 ||x||_2^2, where lambda > 0 is the regularization parameter controlling how hard you push toward small x. This has a clean closed form: the regularized normal equations (A^T A + lambda^2 I) x = A^T b. Adding lambda^2 I to A^T A lifts every eigenvalue by lambda^2, so the matrix is always invertible and its conditioning is bounded no matter how singular A was — that is the whole point. In SVD terms, each singular value's contribution gets multiplied by a filter sigma_i^2 / (sigma_i^2 + lambda^2), which is near 1 for large singular values (kept) and near 0 for tiny ones (suppressed), so the noisy small-sigma directions are damped smoothly rather than blowing up.

Choosing lambda is the crux: too small and you barely regularize (back to the unstable fit), too large and you over-smooth and ignore the data (high bias). This is the bias-variance trade-off made explicit — lambda dials between fitting and stability — and tools like cross-validation or the L-curve help pick it. The general form replaces ||x|| with ||L x|| for some matrix L (penalizing roughness or deviation from a prior), and the same idea underlies ill-posed inverse problems everywhere, from image deblurring to tomography. It is the gentle, ubiquitous antidote to ill-conditioning.