representation collapse

Representation collapse is the failure where the encoder learns a degenerate, trivial solution: instead of mapping different images to different, meaningful embeddings, it maps everything to nearly the same useless output. It is the lurking disaster of any method that trains by making positive views agree, because the laziest way to make two views identical is to make all outputs identical. A collapsed model has perfect agreement loss and zero information, the embeddings tell you nothing about the input.

There are two flavours. Complete collapse is when the output is a constant vector regardless of the input, the most extreme form. Dimensional (or partial) collapse is subtler: the embeddings vary, but they all lie within a low-dimensional subspace, so many feature dimensions are wasted and effectively constant; this can occur even when the loss looks fine and only shows up when you inspect the covariance or singular-value spectrum of the embeddings. Both reduce the usable capacity of the representation.

Every self-supervised method without explicit negatives needs a dedicated anti-collapse mechanism, and understanding which one a method uses is the key to understanding the method. Contrastive methods (SimCLR, MoCo) use negative repulsion, you cannot collapse to a constant because that would also pull negatives together, which the loss punishes. BYOL and SimSiam use a predictor head plus stop-gradient and asymmetry. DINO uses centering plus sharpening of the teacher distribution. SwAV uses the Sinkhorn equipartition constraint that forces clusters to be balanced. A separate line of work attacks collapse directly through the embedding statistics: Barlow Twins drives the cross-correlation matrix of two views toward the identity, and VICReg explicitly adds a variance term (keep each dimension's variance above a floor) and a covariance term (decorrelate dimensions) to forbid both complete and dimensional collapse.