PCA via SVD

Principal component analysis asks: along which directions does my data vary the most? Given a cloud of data points, PCA finds an orthogonal set of axes — the principal components — ordered so the first captures the most variance, the second the most of what is left, and so on. Project onto the top few and you keep most of the structure while collapsing the dimension.

The textbook route is via the covariance matrix C = (1/(n-1)) X^T X for centered data X; the principal components are its eigenvectors and the variances are its eigenvalues. The SVD route skips forming C entirely. Take the SVD of the centered data, X = U Sigma V^T. Then the right singular vectors v_i are the principal directions, and the variance along the i-th component is sigma_i^2 / (n - 1).

It works because X^T X = V Sigma^2 V^T is exactly the eigendecomposition of (a scalar multiple of) the covariance matrix: the eigenvectors are the v's and the eigenvalues are the sigma_i^2. So the SVD of X delivers the same principal components as diagonalizing the covariance, plus the scores (projections) U Sigma for free.

Why prefer the SVD? Forming X^T X squares the condition number and can destroy small but real directions of variation in floating point — squaring a singular value of 10^{-8} buries it at 10^{-16}, below machine precision. The SVD operates on X directly and keeps full accuracy. This is why every serious numerical library computes PCA through the SVD, not through the covariance matrix.