normal equations
The normal equations are the precise algebraic answer to the least-squares question: given an overdetermined system — more equations than unknowns, with no exact solution — what choice of unknowns comes closest? They are the linear system you must solve to obtain the best least-squares fit, and they fall straight out of setting the gradient of the squared error to zero.
Write the model as A x = b, where A is the matrix of inputs, x the unknown parameters, and b the observed data; because there is no exact solution we minimize the squared residual, the squared length of A x - b. Differentiating that quadratic and setting the gradient to zero gives A transpose times A times x = A transpose times b. These are the normal equations. The name normal is geometric: the residual vector A x - b is made perpendicular (normal) to the column space of A — the best fit is the orthogonal projection of the data onto the space the model can reach, and the leftover error sticks out at a right angle. When A transpose times A is invertible (the columns of A are independent), there is a unique solution.
The normal equations are the backbone of regression and data fitting throughout science. Their honest caveat is numerical: forming A transpose times A squares the condition number of A, which can amplify rounding error badly for ill-conditioned problems. For this reason, careful software usually does not solve the normal equations directly but uses a QR factorization or singular value decomposition instead — mathematically the same answer, numerically far more stable. The normal equations remain, however, the clearest way to see why least squares works and what best is projecting onto.
To fit y = m x + b through points, the normal equations become two equations: sum(x_i^2) m + sum(x_i) b = sum(x_i y_i) and sum(x_i) m + N b = sum(y_i), with N the number of points. Solving this 2 by 2 system gives the best-fit slope and intercept directly.
The least-squares fit reduces to one small linear system — the normal equations — solved exactly.
Forming A transpose times A squares the condition number and can wreck accuracy on ill-conditioned data; production code prefers a QR or SVD solve, which gives the same least-squares answer with far better numerical stability.