JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Aligning Vision and Language: Contrastive Pretraining

Two encoders, one shared space, and a loss that pulls matching image-text pairs together. The foundation every modern VLM stands on.

Two towers, one meeting place

You already know how to embed an image and how to embed a sentence. The question that launched modern multimodal AI is deceptively simple: *can we put both into the same space, so that a photo of a corgi lands near the words "a corgi"?* That is the whole premise of vision-language pretraining. We train two encoders — a vision tower (often a ViT) and a text tower — and ask only that their outputs agree when the image and the caption describe the same thing.

This dual-encoder design is what made CLIP famous. Crucially, the two towers never talk to each other during the forward pass — they only meet at the very end, through a similarity score. That keeps inference cheap (you can pre-compute every image embedding once) and makes retrieval trivial: to find the best caption, just take the nearest text vector.

The two towers never interact until the very end, where a dot product collapses each image and text embedding into a single similarity score.

Diagram of the dot product of two vectors as their cosine-scaled similarity.

The contrastive objective

The engine is a symmetric image-text contrastive loss. Take a batch of N image-caption pairs. Compute all N×N similarities between every image and every caption. The N pairs on the diagonal are positives; the N²−N off-diagonal pairs are negatives. The loss — a cross-entropy over rows and over columns — pushes each positive above all the negatives in its row and its column. This is exactly the InfoNCE objective you met in contrastive learning, now applied across modalities instead of across augmentations.

\mathcal{L}=-\frac{1}{2N}\sum_{i=1}^{N}\left[\log\frac{e^{\langle I_i,\,T_i\rangle/\tau}}{\sum_{j=1}^{N}e^{\langle I_i,\,T_j\rangle/\tau}}+\log\frac{e^{\langle I_i,\,T_i\rangle/\tau}}{\sum_{j=1}^{N}e^{\langle I_j,\,T_i\rangle/\tau}}\right]

The symmetric image-text contrastive loss: an N-way softmax in each direction whose only positive is the true pair, sharpened by the temperature tau.

img_e = l2norm(vision(images))     # (N, d)
txt_e = l2norm(text(captions))     # (N, d)
logits = img_e @ txt_e.T / tau     # (N, N), tau = temperature
labels = arange(N)                 # positives are on the diagonal
loss = (ce(logits, labels) + ce(logits.T, labels)) / 2
Symmetric contrastive loss: each image's positive is its own caption; everything else in the batch is a negative.

What the data buys you: zero-shot transfer

Trained on hundreds of millions of noisy web image-caption pairs, the model learns an open-vocabulary alignment that no fixed label set could give you. To classify an image with no task-specific training, you write the candidate classes as sentences ("a photo of a {cat}"), embed them with the text tower, and pick the nearest. This is zero-shot vision-language classification, and it works because the embedding space encodes meaning, not a closed taxonomy.

  1. Write each class as a natural-language prompt; ensembling several prompt templates per class measurably improves accuracy.
  2. Embed all class prompts once with the text tower and cache them as a classifier weight matrix.
  3. At inference, embed the image and take the dot product against every class vector — softmax gives calibrated-ish scores.
\hat{y}=\arg\max_{c}\ \frac{\langle I,\,T_c\rangle}{\lVert I\rVert\,\lVert T_c\rVert}

Zero-shot classification picks the class whose text-prompt embedding has the highest cosine similarity to the image embedding.

The modality gap

Here is a surprise that humbles the "shared space" story. If you actually plot the image embeddings and text embeddings of a trained CLIP, they do not interleave — they sit in two distinct cones, separated by a clear margin. This is the modality gap. Contrastive training only needs matching pairs to be relatively closer than mismatches; it never forces images and text to occupy the same region absolutely. The gap is partly seeded at initialization and preserved by the loss.

What contrastive alignment cannot do

A dual encoder gives you a global match score, but no fine-grained reasoning. It cannot tell you where the corgi is, count the legs, read a paragraph in the image, or answer "is the dog left or right of the cat?". Its language side is a shallow encoder, not a generator — it cannot produce a sentence at all. To get description, dialogue, and reasoning, we need to connect a powerful language model to the vision features. That bridge is the subject of the next guide.