simclr

SimCLR is a deliberately simple contrastive recipe that, by combining a few ingredients well, first showed self-supervised features rivalling supervised ones on ImageNet. The pipeline is: take a batch of images; for each image create two augmented views; pass all views through a shared base encoder (a ResNet) to get representations; pass those through a small projection head (a 2-layer MLP) to get the vectors where the loss is computed; and apply a contrastive loss that pulls the two views of the same image together and pushes all other views apart.

Three findings made it work and are its lasting lessons. First, the composition and strength of data augmentation matters enormously, the winning combination is aggressive random resized cropping plus colour distortion plus Gaussian blur; remove colour jitter and the model cheats using colour histograms. Second, adding a nonlinear projection head between the representation and the loss substantially improves the representation, and crucially you discard the projection head after pretraining and keep only the encoder output for downstream use. Third, contrastive learning benefits from many negatives, which SimCLR obtains from large batch sizes (up to 4096), since the negatives for each positive are simply the other 2(N-1) views in the batch.

The loss is NT-Xent (normalized temperature-scaled cross entropy), an instance of InfoNCE in which embeddings are L2-normalized and similarities are scaled by a temperature before a softmax. SimCLR's main practical cost is that needing large batches demands a lot of accelerator memory, which is exactly the pain point MoCo's memory queue was designed to remove; SimCLR v2 later showed the same approach scales well to bigger ResNets and supports a semi-supervised distill step.