contrastive learning

Contrastive learning shapes an embedding space by comparison: things that should be similar are pulled together, and things that should be different are pushed apart. In the most common self-supervised version, called instance discrimination, every individual image is treated as its own class. You make two randomly augmented copies of one image (say a random crop with colour jitter) and declare them a positive pair that must land near each other, while every other image in the batch is a negative that must be pushed away.

Geometrically, the encoder maps each view to a point, the points are usually L2-normalized so they live on the surface of a unit hypersphere, and the loss simultaneously increases alignment (positives close) and uniformity (the whole set spread out over the sphere). Pushing negatives apart is the crucial ingredient: without that repulsion the easiest way to make positives agree is to map every image to the same constant point, which is a collapsed, useless solution. Negatives provide the force that keeps the space from collapsing.

The dominant loss is InfoNCE, which frames the comparison as a classification problem: among one positive and many negatives, identify which one is the true match. Because the quality of the signal grows with the number of negatives, the two famous frameworks differ mainly in how they supply lots of negatives cheaply: SimCLR uses very large batches so the negatives are the other images in the same batch, while MoCo keeps a memory queue of recently seen embeddings produced by a slowly-moving momentum encoder. Contrastive methods produce features with strong linear separability and were the breakthrough that first matched supervised pretraining on ImageNet.