singular values

Singular values are the non-negative numbers sigma_1 >= sigma_2 >= ... >= 0 sitting on the diagonal of Sigma in the SVD A = U Sigma V^T. The simplest way to feel what they are: a matrix takes the unit sphere and maps it to an ellipsoid (in the output space); the singular values are exactly the lengths of that ellipsoid's semi-axes. The largest singular value is how much the matrix can stretch any unit vector; the smallest is how much it can shrink one. They are the matrix's amplification factors, sorted from biggest to smallest.

Each singular value carries concrete meaning. sigma_1 is the matrix 2-norm ||A||_2 — the maximum stretch. The number of nonzero singular values is the rank of A; in floating point the number ABOVE the noise floor is the numerical rank. The ratio sigma_1/sigma_min (smallest nonzero) is the 2-norm condition number kappa_2(A), the single most important number for predicting how much error a linear solve or least-squares fit will amplify. A singular value near zero means A nearly collapses some direction — it is almost rank-deficient, almost not invertible. And the singular values relate to eigenvalues through A^T A: sigma_i = sqrt(lambda_i(A^T A)). For a symmetric positive-definite matrix the singular values equal the eigenvalues; in general they do not (singular values are always real and non-negative; eigenvalues can be complex or negative).

Why care so much? Because the SORTED list of singular values, the singular spectrum, tells you how much genuine information a matrix holds. If they decay quickly — sigma_1, sigma_2 big, the rest tiny — the matrix is essentially low-rank and compressible: a few singular triplets capture almost everything (this is the basis of compression, PCA, and model reduction). If they decay slowly, the data is genuinely high-dimensional. And critically, singular values are PERFECTLY conditioned (like symmetric eigenvalues): perturbing A by epsilon changes each singular value by at most epsilon. So unlike the eigenvalues of a nonsymmetric matrix, singular values are always safe to compute and trust.