VICReg (variance-invariance-covariance regularization)

VICReg makes the recipe for non-collapsing self-supervision explicit by writing the loss as three named ingredients. Invariance pulls the two views of an image together; variance keeps each embedding dimension from shrinking to a constant; covariance keeps the dimensions from carrying the same information. Together they say, in plain terms: agree on what the augmentations should not change, but stay spread out and decorrelated.

The variance term applies a hinge on the per-dimension standard deviation across the batch, penalizing it whenever it falls below a target (typically one), which is the explicit anti-collapse force. The covariance term penalizes the squared off-diagonal entries of each branch's covariance matrix, decorrelating features. The invariance term is the plain mean-squared distance between the two embeddings. A weighted sum of the three, applied to both branches, is the whole objective — no negatives, no momentum, no stop-gradient.

Because the three terms are computed independently per branch (variance and covariance never compare the two networks), VICReg does not require the two branches to be identical or even share an architecture, making it convenient for multimodal pairs. It is best understood as a sibling of Barlow Twins: both prevent collapse by regularizing the statistics of the embedding rather than by contrasting against negatives.