low-rank factorization

A big weight matrix often does less work than its size suggests — its information lives in a few dominant directions. Low-rank factorization replaces a dense d-by-k matrix with the product of two thin matrices, d-by-r and r-by-k, where the rank r is much smaller, cutting both the parameter count and the multiply-adds from d·k down to r·(d+k).

The classic route is a truncated SVD of a trained weight matrix, keeping the top r singular directions, after which the factorized layer is fine-tuned to recover accuracy. Related ideas include Tucker and CP decompositions for convolution tensors, and, on the training side, parameterizing the update as low-rank — the LoRA family — so that only the small factors are learned. How much rank can be removed depends on the matrix's singular-value spectrum, which varies sharply from layer to layer.

Low-rank methods compress and accelerate and compose well with quantization, but a globally low rank is a strong assumption: many transformer weight matrices are closer to full rank, so naive factorization can hurt unless the rank is chosen per layer.