whitening for self-supervision

Whitening-based methods prevent collapse the way a statistician would: by force-standardizing the batch of embeddings so they have zero mean, unit variance, and zero correlation between dimensions — a whitened distribution. Once the embeddings are whitened, it is mathematically impossible for them to all collapse onto one point or into a low-dimensional subspace, so you can then safely pull positive pairs together without any negatives at all.

Concretely, methods like W-MSE compute the batch's covariance, find a whitening transform (for example via Cholesky decomposition of the inverse covariance), apply it to the embeddings, and only then minimize the mean-squared distance between whitened positive pairs. The whitening operation is differentiable and built into the forward pass, so it acts as a hard constraint each step rather than a soft penalty. This places whitening in the same information-maximization family as Barlow Twins and VICReg, but it enforces decorrelation exactly through a linear transform instead of approximately through a loss term.

The practical caveats are statistical. The covariance must be estimated from the current batch, so whitening needs reasonably large batches to be stable, and computing or inverting the covariance adds cost and can be numerically delicate. In exchange you get a clean, negative-free objective with a structural — not merely encouraged — guarantee against collapse.