dimensional collapse

Dimensional collapse is the subtle cousin of complete collapse. In complete collapse a model maps every input to the same vector — easy to detect because performance is zero. In dimensional collapse the embeddings still vary across inputs, so loss curves and even some metrics look healthy, yet all the variation is confined to a low-dimensional subspace of the embedding space; most directions carry no information at all. The representation is secretly far smaller than its nominal dimension.

You diagnose it by looking at the singular value spectrum of the embedding covariance matrix: if many singular values are essentially zero, the embeddings live on a thin sheet rather than filling the space. In contrastive and non-contrastive self-supervision this happens for several reasons — strong augmentation can wipe out the directions that distinguish images, and implicit regularization in the optimization can shrink certain directions. It is precisely the failure mode that Barlow Twins, VICReg, and whitening were designed to fight, since their decorrelation and variance terms directly push the spectrum to be flat and full-rank.

The practical lesson is that low training loss is not proof of a good representation. Because the projection head can mask collapse in the loss space while the backbone underneath is impoverished, monitoring the rank or effective dimensionality of the features — not just the loss — is the honest way to catch it.