left and right singular vectors

Every singular value sigma_i comes with two companion directions: a right singular vector v_i in the input space and a left singular vector u_i in the output space. The right vectors are the special input directions that A maps cleanly onto stretched output directions; the left vectors are where they land. The defining relation is A v_i = sigma_i u_i — apply A to v_i, and you get u_i scaled by sigma_i, with no rotation mixing in.

Both families are orthonormal. The right singular vectors v_1, ..., v_n form an orthonormal basis of R^n (the input space); the left singular vectors u_1, ..., u_m form an orthonormal basis of R^m (the output space). The v_i are the eigenvectors of A^T A, the u_i are the eigenvectors of A A^T, and the two are tied together by u_i = (1/sigma_i) A v_i whenever sigma_i is nonzero.

This is the secret behind the SVD's power: A becomes simple once you use these special bases. In the v-basis on the input and the u-basis on the output, A does nothing but scale coordinate i by sigma_i — every messy off-diagonal interaction disappears. The transformation reads as rotate (line up with the v's), stretch (by the sigma's), rotate (line up with the u's).

Caveat: when a singular value repeats, its singular vectors are only determined up to a rotation within that shared subspace, just as repeated eigenvalues give eigenspaces rather than single eigenvectors. And when sigma_i = 0, the corresponding v_i sits in the null space of A — A sends it to zero — while extra u's span the part of the output space A can never reach.