least-squares regression

Least-squares regression draws the best straight relationship through noisy data. You cannot usually fit a line through every point, so instead you pick the coefficients that make the total squared vertical error as small as possible. Geometrically you are projecting the data onto the space your model can reach.

The setup is an overdetermined system A x = b (more equations than unknowns, no exact solution). The least-squares solution minimizes ||A x - b||^2, and calculus gives the normal equations A^T A x = A^T b. When the columns of A are independent, A^T A is invertible and x = (A^T A)^-1 A^T b. The fitted values A x are exactly the orthogonal projection of b onto the column space of A.

The clean geometry is orthogonality: the residual b - A x is perpendicular to every column of A, which is precisely what A^T (b - A x) = 0 says. That perpendicularity is why the squared error is minimized, and it ties regression directly to projections and inner products.

Why it matters: least squares is the single most used fitting tool in all of science and engineering. The practical caveat is conditioning: forming A^T A squares the condition number and can lose accuracy, so for reliable computation prefer the QR decomposition, or the SVD when A is rank-deficient or nearly so (which yields the pseudoinverse / minimum-norm solution).