low-rank factorization

A big weight matrix often carries less information than its size suggests — its rows and columns are correlated, so it is close to a lower-rank matrix. Low-rank factorization approximates a large m×n matrix W by the product of two thin matrices, U (m×r) and V (r×n), with the rank r much smaller than m and n. Instead of m·n numbers and m·n multiplies you now have only r·(m+n) of each, a big saving when r is small — and the layer's behavior is nearly unchanged.

The optimal rank-r approximation in the least-squares (Frobenius) sense comes from the truncated singular value decomposition: keep the r largest singular values and their vectors, by the Eckart–Young theorem. For convolutions the idea generalizes — a k×k×C_in×C_out kernel can be factored into low-rank or separable pieces (for example a 1×1 channel-mixing conv followed by a k×k spatial conv, or a spatially separable k×1 then 1×k), each cutting FLOPs. Tucker and CP tensor decompositions extend this directly to the 4-D convolution tensor.

This is a classic way to compress fully-connected layers and early CNNs, and the same principle is baked into modern architectures: ResNet's bottleneck block uses 1×1 "squeeze" convolutions, and MobileNet's depthwise-separable convolution is essentially a structured low-rank factorization of a standard conv. The idea also powers LoRA, which adapts a large pretrained vision or language model cheaply by adding a learned low-rank update to frozen weights. Factorized models usually need a short fine-tune to absorb the approximation error.