orthogonal projection onto the column space

Here is the picture that makes least squares feel inevitable. The vectors A x, as x ranges over everything, sweep out a flat subspace — the column space (range) of A, sitting inside the bigger space where b lives. Our target b is, in general, not in that subspace. The closest point of the subspace to b is its shadow: drop a perpendicular from b straight down onto the subspace and where it lands is the nearest achievable A x. That foot-of-the-perpendicular is the orthogonal projection of b onto the column space, and it is exactly what least squares finds.

Why must the closest point be the perpendicular foot? Because for any other point in the subspace, the line from b to it is slanted, and a slanted segment to a flat surface is always longer than the perpendicular one (Pythagoras). So the minimizer x is the one for which A x = (the projection of b), and the leftover residual r = b - A x sticks straight out of the subspace — it is orthogonal to every column of A. Writing that orthogonality out, A^T (b - A x) = 0, is literally the normal equations A^T A x = A^T b. The geometry and the algebra are the same fact seen from two sides.

This projection viewpoint explains everything downstream: the residual being orthogonal to the columns means it is uncorrelated with each predictor (a fact regression diagnostics rely on); the projection is unique even when x is not (rank-deficient A); and if P is the projection matrix onto the column space, then A x = P b. The map b -> P b is idempotent (P^2 = P, projecting twice changes nothing) and symmetric (P^T = P, the hallmark of an ORTHOGONAL rather than oblique projection).