contrastive image-text pretraining

Contrastive image-text pretraining is the training strategy behind CLIP and its successors: take millions to billions of image-caption pairs and teach two encoders to place matching pairs near each other in a shared embedding space while pushing non-matching pairs apart. The word 'contrastive' is the key idea — the model learns not by predicting exact pixels or exact words, but by contrasting a correct pairing against many wrong pairings and learning what separates them. This sidesteps the need for hand-labeled categories: the caption that already accompanies a web image is the supervision.

The standard loss is InfoNCE applied symmetrically. Within a batch of N pairs, treat each image's true caption as the single positive and the other N−1 captions as negatives; do a softmax over the similarity scores so the model is rewarded for ranking the positive highest, and do the symmetric thing for each caption over images. Similarities are cosine similarities divided by a temperature τ (a learned scalar that controls how sharply the model separates positives from negatives). Because the negatives come from the same batch ('in-batch negatives'), a larger batch means more negatives and a better signal — which is why these models are trained with very large batches across many accelerators.

Beyond CLIP, key variants include ALIGN (Google), which showed the recipe works on a billion noisy, unfiltered alt-text pairs, trading data cleanliness for scale; and SigLIP (Google), which replaces the softmax/InfoNCE with a per-pair sigmoid loss, removing the need to normalize over the whole batch and enabling efficient training at smaller batch sizes. The general lesson — align two modalities by contrasting matched against unmatched at scale — has become a foundational pretraining paradigm for multimodal AI.