the linear least-squares problem

When you cannot satisfy every equation, what is the next best thing? The least-squares answer is: choose x so that the leftover, the residual r = b - A x, is as small as possible — and 'small' is measured by the ordinary Euclidean length. You are picking the x whose predictions A x come closest, overall, to the observed data b. It is the workhorse behind drawing the best straight line through scattered points and behind a huge amount of statistics and machine learning.

Precisely: given an m-by-n matrix A and a vector b, find x that minimizes ||A x - b||_2, equivalently the sum of squared residuals (A x - b)_1^2 + ... + (A x - b)_m^2. Squaring the residuals is the key choice: it makes the objective a smooth, bowl-shaped (convex quadratic) function of x, so there is a unique minimum whenever A has full column rank (its columns are independent), and we can find it by setting the gradient to zero. Why squares and not, say, absolute values? Squares give a clean closed-form linear solution, weigh big misses heavily, and — under Gaussian noise — match the maximum-likelihood estimate. The downside: squaring also makes least squares sensitive to outliers, since one wild point contributes its error squared.

There are several numerically distinct ways to actually compute the minimizer: form and solve the normal equations A^T A x = A^T b (cheap but squares the conditioning), use a QR factorization of A (the standard, stable choice), or use the SVD / pseudoinverse (most robust, and the right tool when A is rank-deficient). They all aim at the same x but differ sharply in accuracy on ill-conditioned data. 'Least squares' done well is one of the quiet triumphs of numerical computing.