Courant-Fischer min-max theorem

The Rayleigh quotient pins down the largest and smallest eigenvalues as a max and a min. The Courant-Fischer theorem completes the picture by giving every eigenvalue in between its own variational formula. The trick is to optimize the Rayleigh quotient not over the whole space but over subspaces of a chosen dimension, and then optimize again over which subspace you pick.

Order the eigenvalues of a self-adjoint T as lambda_1 >= lambda_2 >= ... >= lambda_n. The k-th eigenvalue is the min-max lambda_k = min over (n-k+1)-dimensional subspaces W of [ max over nonzero v in W of R(v) ]. There is a mirror max-min form: lambda_k = max over k-dimensional subspaces U of [ min over nonzero v in U of R(v) ]. Either way, you sweep over subspaces, take an extreme Rayleigh quotient inside each, then take the opposite extreme over all subspaces. The optimal subspace is spanned by the appropriate eigenvectors.

The power of this is that it characterizes lambda_k without referring to the other eigenvectors at all — just dimensions and inner products. That makes it the standard tool for comparing spectra of different operators. It immediately yields the Weyl inequalities for sums, the Cauchy interlacing theorem for principal submatrices, and monotonicity results: if T <= S in the positive-definite order then lambda_k(T) <= lambda_k(S) for every k. Many perturbation bounds in numerical linear algebra are one or two lines from Courant-Fischer.