clip

CLIP (Contrastive Language-Image Pre-training), released by OpenAI in 2021, is a vision-language model that learns to tell which caption goes with which image. It is built from two separate encoders — an image encoder (a ResNet or, in the strong variants, a Vision Transformer) and a text encoder (a Transformer) — that each output a single vector. CLIP is trained so that an image's vector and its true caption's vector are similar, and dissimilar to all the other captions in the batch. The clever part is that natural-language captions, rather than fixed class labels, act as the supervision, so the model learns an open-ended notion of visual concepts.

The training objective is contrastive: in a batch of N image-text pairs, the model forms an N×N matrix of cosine similarities between every image and every text, then is trained (with a symmetric cross-entropy loss, scaled by a learned temperature) to make the N correct pairs the largest entries in their row and column. With a batch of thousands, each image is contrasted against thousands of wrong captions at once, which is what makes the learned space so discriminative. CLIP was trained on roughly 400 million image-text pairs scraped from the web (the WIT dataset).

CLIP's headline ability is zero-shot recognition. To classify an image into categories it was never explicitly trained on, you write each category as a text prompt ('a photo of a cat', 'a photo of a dog'), embed those prompts with the text encoder, and pick the one whose vector is most similar to the image's vector. Because of this, CLIP became the default backbone for a huge range of later systems: it provides the vision encoder and the aligned text space used by open-vocabulary detection, text-to-image generation guidance, the CLIPScore caption metric, and the visual encoders inside many multimodal LLMs.