Contrastive Learning: Pull Together, Push Apart

The core intuition: same thing, different views

In Guide 1 we agreed on the dream: learn good features from raw images, with no human labels. This guide delivers the first method that actually works. Start with one photo — say, a picture of a corgi. Now make two random edits of it: crop a piece and zoom in, shift the colors a little, maybe flip it left-to-right. Call each edited copy a view. The two views look different pixel-by-pixel, but they show the same dog. The whole idea is this: a good network should map both views to nearby points in its feature space, because they share content — while views of a totally different image (a cat, a car, a tree) should land far away.

Here is the everyday analogy your brain already runs. You recognize a friend whether you see them face-on, from the side, half in shadow, across a noisy room, or wearing a new hat. The retinal image is wildly different each time, yet you collapse all of them onto one stable idea: that person. You also keep them firmly distinct from strangers. Self-supervised vision wants exactly that reflex — a representation that says 'same identity' across superficial changes and 'different identity' across genuinely different things.

Contrastive learning is the technique that turns this reflex into a trainable objective. Its principle is two short verbs: agree across views, disagree across instances. We train the network to make two views of the same image agree (their feature vectors close), and views of different images disagree (their vectors far). Crucially, the 'sameness' the network must respect is defined entirely by which edits we allow — that property is called augmentation invariance, and the next section shows it is the secret engine of the whole method.

Augmentations build invariance

An augmentation is just a random, content-preserving edit applied to an image. The standard contrastive toolkit is small and worth memorizing: random resized crop (cut out a random patch and rescale it), color jitter (randomly nudge brightness, contrast, saturation, hue), Gaussian blur (soften detail), random grayscale (drop color entirely with some probability), and horizontal flip (mirror left-right). To make a positive pair, we draw two independent random pipelines from this set and run the same source image through both. The two views differ in crop, color, sharpness — but never in 'what the photo is of.'

Now the load-bearing insight, and it surprises almost everyone the first time: the set of augmentations IS the supervision signal. Think about what you are literally asking the network to do. You demand that the two views map to the same vector. The two views differ only by whatever the augmentations changed. So to satisfy you, the network must produce features that do not change when those things change — it must become invariant to them. Choose to jitter color, and you are commanding 'exact color must not matter.' Choose to crop, and you command 'absolute position and scale must not matter.' Augmentation invariance is therefore not a side effect; it is the precise definition of what 'same content' means in contrastive learning. What you augment is exactly what the model learns to ignore.

A concrete worked example makes this unforgettable. Suppose you are lazy and use only cropping. There is now a cheap shortcut the network can exploit: two crops of the same photo tend to have a similar average color (a beach photo is blue-ish all over, a forest greenish). The network can satisfy 'make the two views agree' by computing little more than the mean RGB of each crop — it never has to understand shape, texture, or objects. The features it learns are nearly useless. Now add color jitter: the two views of the beach now have deliberately mismatched colors, so average color is no longer a reliable cue for 'same image.' The cheat is destroyed. To still match the pair, the network is forced onto something color-invariant and far more meaningful — shape, layout, the actual content. The augmentation didn't just add variety; it closed an escape route.

Positives and negatives

Time to nail down the vocabulary precisely, because the rest of the field uses it constantly. A positive pair is two augmented views of the same source image; they should attract — be pulled close in feature space. A negative pair is two views that come from different source images; they should repel — be pushed apart. These two relations, positives and negatives, are the entire input to the learning signal. Notice nothing here required a class label: 'positive' just means 'same origin photo,' a fact we generated ourselves the instant we chose which crops to pair.

A physical metaphor makes the dynamics vivid. Imagine every view as a little bead floating in feature space. Between each positive pair we hang a spring that pulls them together; between an anchor and each negative we place a magnet that repels. At the start the beads sit randomly. As training applies these forces over and over, the beads slide around until the springs and magnets reach a truce: views of the same image clump into tight knots, and different images settle into separate regions. Nobody told the network what a corgi is — yet 'corgi-ish' views drift together simply because they kept being each other's positives, and the space self-organizes into meaningful neighborhoods.

Why bother with the repulsive negatives at all — why not just pull positives together and be done? Because the pull alone has a catastrophically easy cheat. If the only rule is 'make matching views close,' the network can win instantly by mapping every image to the exact same vector. Then all positives are distance zero — perfect score — and the features carry no information whatsoever. This failure is called collapse, and the negatives are precisely the cure: the repulsive force forbids the all-to-one-point solution by insisting different images stay apart, so the space cannot deflate. Collapse is the central villain of this whole track; contrastive learning beats it with negatives, and Guide 4 will show the clever methods that survive without any negatives at all.

One practical surprise that makes the bookkeeping painless: in practice we do not hunt for negatives anywhere. We process a large minibatch of images at once, and for any given anchor view, all the other images in that same batch are simply treated as its negatives. They came from different photos, so by definition they are valid negatives — free of charge, no extra data loading. This is why batch size will matter so much later: a bigger batch hands you more negatives per step, and (as the next section's loss makes clear) more negatives generally means a stronger, more discriminating signal.

The InfoNCE loss, term by term

We have an intuition (springs and magnets) but a network only learns from a loss — a single number to push down. So we need to convert 'pull positives close, push negatives away' into one differentiable formula. The cleanest way is to reframe the whole thing as a classification problem. Stand at one view, the anchor. In front of you is a lineup: its one true positive partner, plus a crowd of negatives. The task: point to the positive. If the network can reliably pick the right partner out of the lineup, it must have learned features where same-content views are closer than everything else. That is exactly the behavior we want, and 'pick the correct one out of many' is a problem we already know how to score — softmax classification.

First we need a way to measure 'closeness' between two feature vectors. We use cosine similarity: the dot product of the two vectors after each has been scaled to unit length. It reads off the angle between them — ignoring how long the vectors are and caring only about direction.

Cosine similarity: 1 means same direction, 0 means perpendicular, −1 means opposite.

Unpacking it: u and v are two feature vectors; u^{\top}v is their dot product (multiply matching components and add them up); \lVert u\rVert is the length (Euclidean norm) of u, and dividing by both lengths removes magnitude so only the angle survives. The output runs from -1 to 1: a value of 1 means the two vectors point the same way (maximally similar), 0 means they are perpendicular (unrelated), -1 means they point opposite. Tiny example: if u=(1,0) and v=(1,1), then u^{\top}v=1, \lVert u\rVert=1, \lVert v\rVert=\sqrt2\approx1.41, so \mathrm{sim}=1/1.41\approx0.71 — a 45° angle, fairly similar. In SimCLR the embeddings are L2-normalized before the loss, so 'similarity' is literally just this cosine.

Now the heart of the guide — the InfoNCE loss (in SimCLR it is also called the NT-Xent loss, for 'normalized temperature-scaled cross-entropy'). For a positive pair of views indexed (i,j), the loss for anchor i is:

InfoNCE / NT-Xent: a (2N−1)-way softmax whose one correct class is the positive view j.

Let us dissect every symbol. N is the number of source images in the batch; because we make two views of each, there are 2N views and 2N embeddings total. z_i is the (L2-normalized) embedding of view i — our anchor — and z_j is the embedding of j, its positive partner, the other view of the same source image. \mathrm{sim}(z_i,z_k) is the cosine similarity we just defined. \tau (tau) is the temperature, a small positive number (often around 0.1–0.5) that we will study closely; dividing every similarity by it before exponentiating sharpens or softens the resulting distribution. The indicator \mathbb{1}_{[k\neq i]} is a switch that equals 1 for every k except k=i and 0 when k=i — it simply removes the anchor's self-comparison from the sum. So the denominator runs over all 2N views except i itself: that is the single positive z_j plus all 2N-2 negatives, i.e. 2N-1 terms in total.

Read the fraction as a softmax probability. The numerator is the 'score' of the correct answer (anchor with its positive); the denominator is the sum of scores over the whole lineup (positive plus every negative). The ratio is therefore the model's estimated probability that it picked the positive out of the 2N-1 candidates. Taking -\log of that probability is exactly cross-entropy: it is near 0 when the model is confident and right (probability near 1), and blows up large when the model puts its mass on a negative. Minimizing \mathcal{L}_i thus does two things at once and inseparably — it pushes the numerator up (pull the positive closer) and the denominator's negative terms down (push negatives away). Springs and magnets, now written as one number a gradient can chew on.

# Tiny worked example: N = 2 images -> 2N = 4 views
# Image A gives views z1, z2 (a POSITIVE pair).
# Image B gives views z3, z4 (another positive pair).
# Take anchor i = 1. Its positive is j = 2; negatives are 3 and 4.
#
# Suppose the cosine similarities of the anchor to the others are:
#   sim(z1, z2) = 0.90   <- positive (same image A)
#   sim(z1, z3) = 0.20   <- negative (image B)
#   sim(z1, z4) = 0.10   <- negative (image B)
# (sim(z1, z1) is EXCLUDED by the indicator 1_{k != i}.)

import math
tau = 0.1                       # temperature

sims = {"pos": 0.90, "neg1": 0.20, "neg2": 0.10}
exp  = {k: math.exp(v / tau) for k, v in sims.items()}
# exp(9.0)=8103.08 ,  exp(2.0)=7.389 ,  exp(1.0)=2.718

denominator = sum(exp.values())             # 8103.08 + 7.389 + 2.718 = 8113.19
prob_positive = exp["pos"] / denominator    # 8103.08 / 8113.19 = 0.99875
loss_i = -math.log(prob_positive)           # 0.00125  -> tiny: anchor already close

print(round(prob_positive, 5), round(loss_i, 5))   # 0.99875 0.00125

Walk the numbers. We divide each similarity by \tau=0.1 (so 0.9 becomes 9.0, etc.), exponentiate, and sum: the denominator 8113.19 is dominated by the positive's 8103.08 because it scored highest. The positive's share is 0.99875, and -\log(0.99875)\approx0.00125 — a near-zero loss, telling us this anchor is already doing its job. Now feel the temperature's effect by re-running with \tau=1 instead of 0.1: the exponents become 0.9,0.2,0.1, giving \exp values 2.46, 1.22, 1.11, a denominator of 4.79, a positive share of only 0.514, and a loss of \approx0.665. Same embeddings, very different loss. A small \tau blows the gaps between similarities up enormously, so the softmax becomes peaky and the loss obsesses over the hardest negatives — the few that are nearly as close as the positive — punishing them fiercely. A large \tau flattens everything, treating all negatives gently and roughly equally. Temperature is the knob that decides how harshly the model focuses on its toughest competitors.

Two housekeeping notes so the loss is symmetric and complete. First, \mathcal{L}_i is written from anchor i's point of view, but the partner j is just as entitled to its own term \mathcal{L}_j (anchor j, positive i); the full batch loss averages \mathcal{L} over all 2N views, so every view takes a turn as anchor. Second, recall that all of this — the InfoNCE loss, its single positive and crowd of negatives, the whole positive-versus-negatives structure — is the concrete machinery that makes contrastive learning trainable. The springs-and-magnets picture and this formula are the same object seen from two angles.

SimCLR: putting it all together

Every part is now in hand, so let us snap together the canonical contrastive system: SimCLR (A Simple Framework for Contrastive Learning of Visual Representations). It has just four moving pieces. (1) An encoder f — a standard backbone such as a ResNet — that turns an image into a representation vector h; this is the thing we actually want to learn. (2) A small projection head g — a tiny MLP (one hidden layer with a nonlinearity) — that maps h to a lower-dimensional vector z, and it is on z that the InfoNCE loss is computed. (3) The augmentation pipeline from Section 2, applied twice per image. (4) A large batch, which supplies the negatives for free. That is the entire apparatus.

1. Sample a minibatch of N images.
2. Augment each image twice with independent random pipelines, producing 2N views (each pair of views from one image is a positive pair).
3. Run all 2N views through the SAME encoder f to get representations h (the weights are shared across all views).
4. Pass each h through the SAME projection head g to get embeddings z, then L2-normalize them to unit length.
5. For every view (as anchor), compute the InfoNCE loss against its one positive and the 2N−2 in-batch negatives; average over all 2N anchors.
6. Backpropagate and update f and g together. Repeat for many epochs.

# SimCLR training step (PyTorch-style pseudocode)
for images in loader:                 # images: a batch of N pictures
    v1 = augment(images)              # N views, random pipeline #1
    v2 = augment(images)              # N views, random pipeline #2
    views = concat(v1, v2)            # 2N views; view k and view k+N are positives

    h = f(views)                      # encoder -> representations,  shape (2N, d_h)
    z = g(h)                          # projection head -> embeddings, shape (2N, d_z)
    z = normalize(z, dim=1)           # L2-normalize so dot product == cosine sim

    sim = (z @ z.T) / tau             # (2N x 2N) all-pairs cosine sims, temperature-scaled
    mask_self = eye(2N)               # marks the k == i diagonal
    sim.masked_fill_(mask_self.bool(), -inf)   # remove self-comparison (the indicator)

    targets = positive_index(2N)      # for row i, the column of its positive partner
    loss = cross_entropy(sim, targets)# (2N-1)-way softmax classification; correct class = positive

    loss.backward(); optimizer.step(); optimizer.zero_grad()

# After training: THROW AWAY g, keep the encoder f for downstream tasks.

SimCLR's lasting lesson was not a new architecture but a precise empirical recipe — small choices that, stacked together, made contrastive learning finally rival supervised pretraining. Four ingredients carried most of the weight: (1) a strong composition of augmentations — crucially color jitter plus random crop together, since either alone leaves an exploitable shortcut, as we saw; (2) the nonlinear projection head g, which improved features markedly over applying the loss to h directly (Section 6 explains the surprising reason); (3) large batch sizes (thousands of images), because the in-batch negatives are the only negatives, so a bigger batch means a richer, harder lineup; and (4) long training with many epochs, because each view needs to meet many different negatives over time to carve out a clean place in the space.

When training ends, here is the move that ties this guide back to Guide 1: we discard the projection head g entirely and keep only the encoder f. The vector h=f(\text{image}) is the general-purpose representation we set out to learn. To measure how good it is, we use the linear evaluation protocol from Guide 1 — freeze f, train a single linear classifier on top of h using whatever labels we have, and read off the accuracy. A high linear-probe number means f learned features so clean that even a one-layer classifier can separate the classes — exactly the payoff we wanted, achieved without a single label during pretraining.

Why temperature and the projection head matter

Two design choices in SimCLR reliably trip up newcomers because their effects are larger and stranger than they look. The first is the temperature \tau. We already saw mechanically that small \tau sharpens the softmax; the consequence is that the gradient concentrates almost entirely on the hard negatives — the handful of other images whose embeddings are nearly as close to the anchor as the true positive. That focus is powerful: hard negatives are the ones that teach the finest distinctions ('this husky is not that malamute'). But push \tau too small and the loss becomes obsessed with a few near-duplicates, the gradients grow spiky and unstable, and useful broad structure gets ignored. Push \tau too large and every negative is treated alike, the signal goes mushy, and features blur together. There is no universal best value; \tau is a genuinely sensitive hyperparameter (commonly tuned in the 0.05–0.5 range) and it materially changes what the InfoNCE loss rewards.

The second choice is the most counterintuitive result in the whole paper: at evaluation you should **throw away the projection head g and use the representation h that comes before it**, not the embedding z the loss was actually computed on. It feels backwards — surely the layer the loss optimized is the best one? The resolution is subtle and worth internalizing. The loss demands invariance: it forces z to forget everything that differed between the two views — color, orientation, exact crop. That is correct for solving the contrastive task, but a lot of that discarded information (color, pose, location) is genuinely useful for downstream tasks. The projection head g acts as a sacrificial buffer: it absorbs the harsh invariances the loss insists on, so that the pre-head representation h can stay richer and more general. In short, the space where the loss lives and the space of broadly useful features are not the same place — so we optimize in one and harvest from the other.

Finally, a short checklist of practical pitfalls, each a direct corollary of what we built. Too-small batches: since negatives come only from within the batch, a small batch gives a thin, easy lineup and a weak signal — this is the headache Guide 3 (MoCo) will solve with a memory queue of negatives. Too-weak augmentation: leave color jitter out and the network rediscovers the average-color shortcut from Section 2; weak augmentation means weak invariance means lazy features. Unnormalized embeddings: if you forget to L2-normalize z before the loss, 'similarity' is no longer a clean cosine, the temperature stops meaning what you think, and training often becomes unstable. Get these three right and a basic contrastive setup will already learn features that, under the linear protocol, come startlingly close to supervised pretraining.