the Moore-Penrose pseudoinverse
/ MOOR PEN-rohz /
A square invertible matrix has a true inverse A^(-1) that undoes it. But most matrices in data work are rectangular or singular, with no honest inverse. The Moore-Penrose pseudoinverse, written A^+, is the best stand-in: a single matrix that, for any A, gives the most sensible 'solve' of A x = b. For a tall full-rank A it hands you exactly the least-squares solution; for a wide one it gives the minimum-length exact solution; and for anything in between it does the closest thing to both. It is the one formula that captures 'solve as well as possible' in all cases.
Its cleanest definition is through the SVD. Write A = U Sigma V^T, where Sigma holds the singular values sigma_1 >= sigma_2 >= ... on its diagonal. Then A^+ = V Sigma^+ U^T, where Sigma^+ is formed by taking the reciprocal 1/sigma_i of each NONZERO singular value (and leaving the zeros as zeros). The least-squares solution is then simply x = A^+ b. For a full-rank tall A this equals (A^T A)^(-1) A^T b — the normal-equations answer — but computing it via the SVD is far more stable, and it keeps working when A^T A is singular. The pseudoinverse is uniquely pinned down by four algebraic 'Penrose conditions', which is why it is THE generalized inverse, not just one of many.
Why care? Because rank-deficient and nearly-singular least-squares problems are everywhere — collinear predictors, more features than samples, redundant sensors. When some sigma_i are zero or tiny, A^+ gives the minimum-norm least-squares solution: among all x that achieve the smallest residual, it picks the shortest one, a sane default. The danger is the tiny singular values: 1/sigma_i explodes when sigma_i is near zero, so a naive pseudoinverse amplifies noise wildly. That is exactly what the truncated SVD and Tikhonov regularization are there to tame — and it is why you should almost never form A^+ explicitly just to solve one system.
If A has SVD with singular values (4, 2, 0.0001), then A^+ uses reciprocals (1/4, 1/2, 10000). The last, from a near-zero singular value, hugely amplifies any noise lying along that direction — a warning that the problem is ill-conditioned and that the tiny value should probably be truncated to zero.
The pseudoinverse inverts only the nonzero singular values; tiny ones blow up noise unless truncated.
Do not form A^+ explicitly to solve one least-squares problem — it is wasteful and less accurate than solving via QR or SVD directly. The pseudoinverse is a conceptual tool; the tiny singular values it inverts are exactly where ill-conditioning hides.