principal component analysis

Imagine a cloud of data points in high-dimensional space. Most of the spread might lie along just a few directions, while other directions barely vary at all. PCA finds those special directions, ranked by how much variance they capture, so you can describe the data with far fewer numbers and lose almost nothing.

Precisely: center the data so each feature has mean zero, form the covariance matrix C, and compute its eigenvalues and eigenvectors. Because C is symmetric and positive semidefinite, the spectral theorem guarantees an orthonormal eigenbasis. The eigenvectors are the principal axes; the eigenvalue lambda_i is the variance along axis i. Keeping the top k eigenvectors projects the data onto the best k-dimensional subspace in the least-squares sense.

Equivalently, run an SVD of the centered data matrix X (rows = samples). If X = U Sigma V^T, the right singular vectors (columns of V) are the principal axes and sigma_i^2 / (n-1) are the variances. The SVD route is numerically preferred because it never explicitly squares the data into C, avoiding loss of precision.

Why it matters: PCA is the default first move for compression, denoising, and visualization. The caveat is that it only sees linear, second-order structure (variance and covariance). If the meaningful structure is curved or non-Gaussian, PCA can mislead, and because it chases variance it is sensitive to feature scaling.