CLIP: Teaching One Model to Match Pictures and Captions

Learning From the Whole Web, No Labels Needed

Imagine you wanted to teach a model to recognise thousands of different things in photos. The old way was painful: hire people to draw boxes and type labels onto millions of images, one category at a time. That is slow, expensive, and locked to a fixed list of classes. But here is the quiet trick that changed everything — the internet has already done the labelling for us. Every photo on a webpage tends to sit next to descriptive text: an alt-text attribute, a caption, a product title, a tweet. Scrape the web and you harvest hundreds of millions of (image, text) pairs essentially for free. Nobody had to sit down and annotate them; the pairing came baked into how people publish images online.

In the previous guide we met the central idea of a shared embedding space — a single coordinate system where a picture of a dog and the words "a dog" can land near each other, so that distance between any image and any text means something. We left it as a promise. This guide is where we actually build that space. The recipe is called contrastive image-text pretraining, and the famous model that popularised it is CLIP (Contrastive Language–Image Pre-training). Think of CLIP as the concrete machine that takes those free web pairs and folds vision and language into one space.

What exactly is the lesson the model studies? Here it is in one sentence: within a batch of image-caption pairs, learn which caption goes with which image. That is the entire supervision signal — no class labels, no bounding boxes, just "these two belong together, those two do not." Because the model has to tell pairs apart from non-pairs, the approach is called contrastive: it learns by contrasting the true match against many wrong ones. The next two sections turn this one-sentence idea into an architecture and then into precise math.

Two Encoders, One Shared Space

CLIP is built from two separate towers that never share weights. The first is an image encoder: a network that reads pixels and outputs a single summary vector for the whole picture. In CLIP this is either a Vision Transformer (the ViT that splits an image into patches and attends over them, which we met in an earlier vision track) or a ResNet (the classic convolutional backbone). The second is a text encoder: a Transformer that reads the caption token by token and outputs a single summary vector for the whole sentence. So a picture becomes one vector, and a sentence becomes one vector — two outputs from two different machines.

There is a problem, though: the image encoder and the text encoder naturally produce vectors of different sizes, living in different coordinate systems — you cannot directly compare them. So CLIP bolts a small linear projection onto each tower that maps its output into the same dimension, say a shared length of 512. Now an image vector and a text vector are both points in the same 512-dimensional space. The final step is L2 normalization: divide each vector by its own length so that every vector sits exactly on the surface of the unit sphere (length = 1). After this, comparing an image to a caption is just measuring the angle between two unit arrows.

The normalized similarity between image i and caption j.

Let us read this symbol by symbol. The term \mathbf{img}_i is the embedding vector of the i-th image in the batch, and \mathbf{txt}_j is the embedding of the j-th caption. The double bars \lVert \cdot \rVert mean "the length (norm) of this vector," so dividing a vector by its own norm shrinks or stretches it to length exactly 1 — that is the normalization step. The dot \cdot is the dot product. Here is the key fact: the dot product of two unit-length vectors is exactly their cosine similarity — a number from −1 (pointing opposite) through 0 (perpendicular, unrelated) to +1 (pointing the same way, a perfect match). So s_{ij} is a single score saying how well image i matches caption j. If you arrange all of them in a grid, the diagonal entries s_{ii} are the true pairs (image i with its own caption i), and everything off the diagonal is a mismatch. Why normalize first? Because without it, a vector could score high just by being long, not by pointing in the right direction — normalization strips out magnitude so that only direction (meaning) counts, putting every comparison on equal footing. That grid of scores, the per-batch similarity matrix, is the object the next section will optimise.

The Contrastive Objective, Step by Step

Now the heart of CLIP. Take a batch of N image-caption pairs. Encode all N images and all N captions, then compute every pairwise score from the last section to fill an N×N similarity matrix. The N entries on the diagonal are the correct matches; every one of the N²−N off-diagonal entries is a distractor — a caption that belongs to some other image. The training goal, intuitively, is to make each diagonal score the brightest in its own row and its own column, while dimming all the distractors around it.

The clever move is to recast this as a familiar classification problem — twice. Look at one row of the matrix: it holds image i's scores against all N captions. Ask "which of these N captions matches this image?" The right answer is caption i. That is exactly an N-way classification problem, and we already know how to train those with softmax + cross-entropy. Now do the same down each column: "which of the N images matches this caption?" Two directions — image→text and text→image — and we simply average their losses. That symmetry is why it is called the symmetric contrastive loss.

The symmetric InfoNCE loss: image→text plus text→image, averaged.

Let us unpack every piece. Inside the sum, s_{ii} is the diagonal score — the correct image-caption match for example i — and the s_{ij} in the denominator run over all N captions, the correct one plus the N−1 distractors. The symbol \tau (tau) is the temperature, a small positive number we divide the scores by before exponentiating. The fraction \exp(s_{ii}/\tau)\big/\sum_j \exp(s_{ij}/\tau) is just a softmax: it turns the row of raw scores into a probability distribution over the N candidates, and we read off the probability the model assigned to the true caption. Taking -\log of that probability is ordinary cross-entropy: if the model gives the true pair probability 1 the loss is -\log 1 = 0 (perfect), and as the probability drops toward 0 the loss shoots up, punishing the model. The -\frac{1}{N}\sum_i averages this over the whole batch. \mathcal{L}_{\text{t2i}} is the identical formula but reading down columns instead of across rows, and the final \mathcal{L} averages the two directions. Temperature controls sharpness: a smaller \tau divides by a smaller number, blowing the score gaps up and making the softmax more peaky and confident; a larger \tau flattens it toward a uniform guess. CLIP actually learns \tau during training. This whole construction is known as the InfoNCE loss.

Let us grind through a tiny concrete example with N = 3 so the arithmetic is visible. Suppose after normalizing and dotting we got this 3×3 similarity matrix (rows = images, columns = captions), and the temperature is τ = 0.5:

# similarity matrix s[i][j]  (diagonal = correct pairs)
#            cap1   cap2   cap3
#  image1 [  0.90   0.20   0.10 ]
#  image2 [  0.30   0.80   0.20 ]
#  image3 [  0.10   0.25   0.85 ]
# temperature tau = 0.5  (so dividing by tau means multiplying by 2)

1. Take row 1 (image 1's scores): [0.90, 0.20, 0.10]. Divide each by τ = 0.5 → [1.80, 0.40, 0.20].
2. Exponentiate each: exp(1.80) = 6.05, exp(0.40) = 1.49, exp(0.20) = 1.22.
3. Sum the denominator: 6.05 + 1.49 + 1.22 = 8.76.
4. Softmax probability of the correct (diagonal) caption: 6.05 / 8.76 = 0.690. The model thinks there is a 69% chance caption 1 matches image 1.
5. Loss for this row: −log(0.690) = 0.371. A perfect 1.0 would give 0; this row is penalised 0.371 for not being fully confident.

# CLIP's symmetric loss in PyTorch-style pseudocode
# image_features: [N, d]   text_features: [N, d]
img = l2_normalize(image_features, axis=1)   # unit-length image embeddings
txt = l2_normalize(text_features,  axis=1)   # unit-length text embeddings

logits = (img @ txt.T) / tau                 # N x N similarity / temperature
labels = arange(N)                           # correct pairs lie on the diagonal

loss_i2t = cross_entropy(logits, labels, axis=1)  # each row  -> its caption
loss_t2i = cross_entropy(logits, labels, axis=0)  # each col  -> its image
loss = (loss_i2t + loss_t2i) / 2             # symmetric average

Zero-Shot Magic: Classifying Unseen Categories

Here is the payoff that made CLIP famous. Once trained, you can ask it to classify an image into categories it was never explicitly trained on — with zero extra fine-tuning. The trick reframes classification as matching. To recognise a photo, write a short text prompt for each candidate class, like "a photo of a cat", "a photo of a dog", "a photo of a car". Embed all those prompts with the text encoder, embed the image with the image encoder, and simply pick the caption whose embedding is closest to the image's. The class of the winning caption is your prediction. This is exactly zero-shot image classification.

Softmax over candidate classes, then pick the most probable.

Reading it symbol by symbol: \mathbf{txt}_c is the text embedding of the prompt "a photo of a {c}" for candidate class c (so \mathbf{txt}_{\text{cat}} is the embedding of "a photo of a cat"). \operatorname{sim}(\mathbf{img}, \mathbf{txt}_c) is the very same cosine similarity from Section 2 — image versus that one label. \tau is the temperature again. The fraction is once more a softmax, but now it runs only over your candidate classes, turning the similarities into probabilities that sum to 1. Finally \arg\max_c just means "return the class c with the highest probability" — the label, not the number. The reason this works on classes CLIP never saw as a label is that the shared space already understands the words: it learned during pretraining where the embedding of the word "cat" lives and what cat-pictures look like nearby, so a brand-new label like "axolotl" still lands somewhere sensible because the text encoder knows that word from the web.

Let us run our cat photo through it with 3 candidate labels and τ = 0.5. Suppose the cosine similarities come out as sim(img, "cat") = 0.40, sim(img, "dog") = 0.15, sim(img, "car") = 0.05. Divide by τ: [0.80, 0.30, 0.10]. Exponentiate: [2.23, 1.35, 1.11], summing to 4.68. The probabilities are 2.23/4.68 = 0.476 for cat, 1.35/4.68 = 0.288 for dog, 1.11/4.68 = 0.236 for car. The argmax is cat at 47.6% — correct, with no training on a "cat vs dog vs car" task at all. (Real CLIP uses a much smaller learned temperature, which would push that 0.476 far closer to 1, i.e. far more confident.)

# Zero-shot classification with CLIP
classes = ["cat", "dog", "car"]
prompts = [f"a photo of a {c}" for c in classes]

txt = l2_normalize(text_encoder(prompts))   # [num_classes, d]
img = l2_normalize(image_encoder(image))    # [d]

sims  = img @ txt.T                          # cosine sim to each class
probs = softmax(sims / tau)
pred  = classes[argmax(probs)]              # -> "cat"

Retrieval: Searching Images With Words

The exact same machinery gives you search. In image-text retrieval you have a big gallery of images and you want to find the ones that match a text query like "a golden retriever playing in snow." Embed that query once with the text encoder to get one vector, then compare it against every image embedding and return the images with the highest cosine similarity. Nothing new is needed — "find the best match" is just "pick the highest cosine similarity," the same comparison CLIP was trained to make.

Crucially this is bidirectional. Text→image gives you "search a photo library by typing a description." Image→text reverses it: hand the model an image and rank a pool of captions to find the one that best describes it. Both directions are the same dot-product comparison, just deciding which side is the fixed query and which is the gallery. The standard scorecard is recall@K: of all the images that truly match a query, what fraction show up somewhere in the top K returned results? Recall@1 is strict (the very first hit must be right); recall@5 or recall@10 is more forgiving and reflects how people actually scan a results page.

Why is the dual-encoder design such a gift in production? Because the image side and the text side are encoded independently, you can embed every image in your gallery once, offline, and store those vectors in a database. At query time you only encode the short text and do fast vector comparisons — you never re-run the heavy image encoder. That precompute-once property is the whole reason CLIP-style models power real search systems.

Strengths, Limits, and What CLIP Can't Do

Time for an honest reckoning, because CLIP's limits are exactly what motivate the rest of this track. Its superpower is global image-text matching: "what is this picture roughly about?" It is fast, scalable, and astonishingly broad. But the very design that makes it fast — squashing each image and each caption into a single vector — is also its ceiling. A single vector is a summary, and summaries throw away detail. So CLIP is reliably weak wherever the answer depends on details the summary discards.

The same blind spot shows up across a family of tasks. Counting: CLIP struggles to distinguish "three dogs" from "five dogs" — the global vector barely encodes exact quantity. Spatial relations: "the cup left of the laptop" versus "right of" is often a coin flip. Fine-grained attributes: telling closely-related bird species or subtle textures apart exceeds what one vector cleanly captures. Reading text inside images (OCR-style): CLIP catches the gist of a sign but cannot reliably read the words on it. None of these are bugs to patch; they are the natural consequence of a single-vector, global-matching design.

One more honest caveat: CLIP learned from raw web data, so it inherits the web's biases — stereotypes in how people caption images, uneven coverage of cultures and groups, and skewed associations. Because nobody curated those hundreds of millions of pairs, the model quietly absorbs whatever correlations the internet happened to contain. Using CLIP responsibly means testing for these biases rather than assuming a model trained on "everything" is therefore neutral.

Each limitation is a doorway to the next guides. To bind attributes and relations correctly, vision and language need to interact deeply instead of meeting only at a final dot product — that is cross-modal attention and fusion, guide 3. To answer "where exactly is the red cube?" the model must point at regions, not just label the whole image — that is grounding, guide 4, which builds on the open-vocabulary idea we teased earlier. And to actually reason — count, compare, explain, follow instructions about an image — we attach vision to a language model that can think step by step, the multimodal LLMs of guide 5. CLIP is the foundation those all stand on; understanding both its power and its single-vector ceiling is exactly what prepares you for what comes next.