the normal equations
How do you actually compute the least-squares solution? The most direct recipe turns one rectangular system into one square system. Multiply both sides of the overdetermined A x = b by A^T (A transpose), and you get A^T A x = A^T b. This square n-by-n system — called the normal equations — has exactly the least-squares x as its solution. 'Normal' here means perpendicular: the equations encode that the residual is normal (orthogonal) to the column space.
Where do they come from? Take the squared-residual objective ||A x - b||_2^2 and minimize it by calculus. Its gradient is 2 A^T (A x - b); setting that to zero gives A^T (A x - b) = 0, i.e. A^T A x = A^T b. The matrix A^T A is n-by-n, symmetric, and (when A has full column rank) positive definite — so you can solve it cheaply with a Cholesky factorization. For a straight-line fit this is the familiar pair of equations relating sums of x_i, y_i, x_i^2, and x_i y_i that you may have seen in a statistics course.
The catch — and it is a serious one — is conditioning. The condition number of A^T A is the SQUARE of the condition number of A: kappa(A^T A) = kappa(A)^2. So if A is even mildly ill-conditioned (say kappa(A) ~ 10^8), forming A^T A pushes it to ~10^16, near the edge of double precision, and you can lose all your accuracy. The normal equations are fast and simple and fine for well-conditioned problems, but for anything delicate the QR or SVD routes — which never form A^T A — are preferred. Knowing WHEN the cheap method is safe is the real skill.
For A with rows (1,1),(1,2),(1,3) and b = (2,2,4)^T, the matrix A^T A has rows (3,6) and (6,14), and A^T b = (8,18)^T. Solving the 2-by-2 system [rows (3,6),(6,14)] x = (8,18)^T gives x = (2/3, 1)^T — the same best-fit line y = 2/3 + x, now obtained by a tiny 2-by-2 solve instead of a rectangular one.
A^T A is small and symmetric — convenient — but its conditioning is the square of A's.
Never form A^T A for an ill-conditioned A: it squares the condition number and can destroy accuracy. The normal equations are convenient, not safe by default — reach for QR or SVD when in doubt.