covariance matrix

A covariance matrix is a compact ledger of how a set of variables move together. The diagonal entry C_ii is the variance of feature i (how much it wiggles on its own); the off-diagonal C_ij is the covariance of features i and j (do they rise and fall together, oppositely, or independently?).

Formally, for centered data with rows as samples, C = (1/(n-1)) X^T X. This matrix is symmetric (C_ij = C_ji) and positive semidefinite: for any direction w, w^T C w = variance of the data projected onto w, which can never be negative. Positive semidefiniteness is exactly why all its eigenvalues are real and nonnegative.

Its eigenvectors are the principal axes of the data and its eigenvalues the variances along them, which is the engine behind PCA. The whole geometry of a Gaussian cloud is captured by C: its eigenvectors orient the ellipsoid of equal density and the square roots of its eigenvalues set the ellipsoid's semi-axis lengths.

A caveat: when you have more features than samples, C is rank-deficient and singular, so some eigenvalues are exactly zero. Estimating C well from limited data is its own art (shrinkage and regularization), because the raw sample covariance is noisy in high dimensions.