Barlow Twins

Barlow Twins takes its name and its principle from neuroscientist Horace Barlow's idea that sensory systems should reduce redundancy. Instead of comparing samples to each other, it compares the dimensions of the embedding to each other. The goal is an embedding whose components are each robust to augmentation yet carry non-overlapping information — invariant but not redundant.

Two augmented views of a batch are encoded and projected, and the empirical cross-correlation matrix between the two view embeddings (computed across the batch, per dimension) is formed. The objective pushes this matrix toward the identity: on-diagonal terms driven to one make each feature invariant to the augmentation, while off-diagonal terms driven to zero decorrelate the features so different dimensions encode different things. A single redundancy-reduction weight λ trades off these two parts.

Its appeal is conceptual economy: no negative pairs, no momentum encoder, no stop-gradient, no predictor, and it is relatively insensitive to batch size. Collapse is prevented structurally — a collapsed constant embedding cannot make off-diagonal correlations vanish — placing Barlow Twins, alongside VICReg and whitening, in the family of information-maximization methods rather than instance-contrastive ones.