weighted least squares

Ordinary least squares treats every data point as equally trustworthy. But often some measurements are sharper than others — a precise instrument here, a noisy one there. Weighted least squares lets you say so: you attach a weight to each equation, large for the measurements you trust and small for the ones you do not, and the fit listens more closely to the reliable points. It is least squares with a sense of priorities.

Concretely, instead of minimizing the plain sum of squared residuals, you minimize a WEIGHTED sum sum_i w_i (A x - b)_i^2, where each w_i > 0 is the weight on equation i. In matrix form this is minimizing (A x - b)^T W (A x - b) with W a diagonal matrix of weights, and the weighted normal equations become A^T W A x = A^T W b. A clean way to implement it is to scale each row of A and each entry of b by sqrt(w_i) — turning the weighted problem into an ordinary least-squares problem on the rescaled data — and then solve that by QR as usual. The statistically principled choice sets each weight to the inverse of that measurement's variance, so noisy points are downweighted exactly in proportion to their uncertainty (this is the heart of the Gauss-Markov optimal estimator).

Weighted least squares matters whenever measurement quality varies — heteroskedastic data (unequal noise), combining sensors of different precision, or giving more recent data more say. The general form, with a full (non-diagonal) covariance matrix in place of W, is called generalized least squares and handles correlated noise. The practical caution is the same as everywhere: choosing weights is a modeling decision; bad weights can fit the noise instead of the signal, and wildly different weights can worsen conditioning.