Scaling Up: MoCo and the Hunt for Negatives

The negatives problem

In the last guide we built a learner that pulls two views of the same image together and pushes views of different images apart. The 'pull together' partners are the positive and negative pairs — specifically the positive pair — and the 'push apart' partners are the negatives. A quiet but stubborn fact runs through all of contrastive learning: the more negatives you can compare each image against, the sharper and more useful the learned features tend to become. With only a handful of negatives, an embedding only has to be different from a few neighbours; with thousands, it has to carve out a genuinely distinctive spot in the representation space.

To see exactly where the negatives live, recall the infonce loss from Guide 2. For a query embedding it forms a softmax in which the numerator is the score of the single correct (positive) match, and the denominator sums the scores of the positive plus every negative. Schematically the loss is the negative log of (positive score) / (positive score + sum over negatives of their scores). The negatives sit entirely inside that denominator. Adding more of them makes the denominator a harder, more competitive bar for the positive to clear — which is precisely why more negatives sharpen the signal the loss sends back through the network.

Now the catch. In simclr, the negatives for any image are simply the other images in the same training batch — there is no separate store of them. So 'more negatives' literally means 'a bigger batch'. SimCLR's best results came from batches of 4096 images or more, with each image producing two views, all encoded and held in memory at once so their gradients can be computed together. That demands a small army of high-memory accelerators (TPUs or top-end GPUs) just to fit one step. For most labs and individuals, that hardware bill is the wall they hit.

A dictionary as a queue

MoCo's first move is a change of perspective. Instead of thinking 'batch of images', think of contrastive learning as a dictionary look-up. You have a query (one view of an image) and you want to find its matching key (another view of the same image) inside a dictionary full of keys. Every key is just a feature vector produced by an encoder. The positive key is the one that belongs to the query; all the other keys in the dictionary are negatives. Learning means tuning the encoder so each query's vector lands closest to its own key and far from the rest.

Here is the key insight. If the dictionary is a separate structure rather than the current batch, its size no longer has to equal the batch size. Momentum Contrast (MoCo) implements the dictionary as a fixed-size queue — a rolling buffer of recently-encoded key vectors. At every training step you encode the current small batch, push (enqueue) its keys onto the queue, and drop (dequeue) the oldest keys to keep the length fixed. After many steps, the queue holds thousands of keys accumulated over the recent past, even though each individual step only ever encodes a small batch. These queued keys serve as the negatives for the current queries.

An analogy makes the rolling buffer intuitive. Imagine a security guard who can only glance at a few faces at a time, but keeps a running gallery of the faces seen over the last few minutes. To decide whether a new face is unfamiliar, they compare it against that whole recent gallery — not against the entire crowd at once, which no one could hold in their head. The gallery is the queue: a manageable working memory of 'recently seen others' that gives every new face plenty to be contrasted against, without ever needing to see everyone simultaneously.

The consistency problem and the momentum encoder

The queue idea has a hidden flaw, and finding it is what makes moco clever. The keys in the queue were not all encoded at the same time — the oldest were produced thousands of steps ago, the newest just now. But the encoder is being trained, so its weights change at every step. If the encoder changes quickly, then a key from 1000 steps ago was made by a noticeably different network than a key from this step. The query is then being compared against keys that speak slightly different 'languages'. This inconsistency makes the comparison noisy and the gradient unreliable.

This is more than a nuisance — it is a stability risk of the same family as the collapse worries that haunt self-supervised learning. If the keys are inconsistent and jittery, the loss can chase a moving target and the features can degenerate rather than improve, drifting toward the trivial solutions that representation collapse describes. A good dictionary must be not only large but also internally consistent: its keys should be comparable to one another, as if drawn from one stable reference.

MoCo's fix is the heart of the method: encode the keys with a SEPARATE encoder — the momentum encoder — whose weights move only very slowly. The query encoder learns fast by ordinary backpropagation; the key encoder is a slow, smoothed echo of it. Because it barely changes from step to step, every key it has ever placed in the queue was made by an almost-identical network, so they all remain mutually comparable. Slowness is consistency. The precise rule that makes the key encoder slow is a single update equation.

The momentum update: the key encoder's weights are an exponential moving average of the query encoder's weights.

Read the equation symbol by symbol. θ_q (theta-q) are the parameters of the QUERY encoder — the network that turns a query view into a vector; these are updated normally, by gradient descent on the loss. θ_k (theta-k) are the parameters of the KEY (momentum) encoder; crucially, gradients NEVER train these directly — they are only ever set by this line. The arrow means 'overwrite the left side with the right side each step'. m (the momentum coefficient) is a number in [0,1), typically 0.999. The formula is an exponential moving average: the new θ_k is mostly its old self (weight m) with a tiny nudge toward the current θ_q (weight 1−m).

Build the intuition with the two extremes. If m = 0, the line becomes θ_k ← θ_q: the key encoder is overwritten to exactly equal the query encoder every single step. That is fast — but it recreates the inconsistency problem, because the key encoder now lurches around just as much as the query encoder. If instead m = 0.999, each step moves θ_k only one part in a thousand toward θ_q, so the key encoder drifts very slowly. A key stored hundreds of steps ago was made by a network only a fraction of a percent different from today's — still mutually comparable. Picture a thoughtful teacher who revises their views gradually rather than overnight, so the notes a student took last month are still consistent with what the teacher says today. That gentle, smoothed drift is exactly what keeps a long queue trustworthy.

Putting MoCo together

We now have all the pieces; let us trace one training step end to end. Take an image and make two augmented views, just like in SimCLR. One view becomes the query and goes through the trainable query encoder to give an embedding q. The other view is the positive key, and a small batch of recent keys plus the whole queue are the negatives — all keys are encoded by the slow momentum encoder. We then apply an infonce loss with one positive and many queued negatives. The gradient flows back only into the query encoder; the momentum encoder is updated solely by the EMA rule from Section 3, never by gradients.

MoCo's InfoNCE loss for one query: identical in form to SimCLR's, but the negatives come from the queue.

Unpack it symbol by symbol. q is the query embedding from the query encoder. k₊ (k-plus) is its single positive key — another view of the same image, produced by the momentum encoder. The k₋ (k-minus) are the many negative keys pulled from the queue, again all from the momentum encoder. The dot q·k measures similarity (with the vectors normalised, this is cosine similarity, as in Guide 2). τ (tau) is the same temperature from Guide 2: a small positive number that scales the scores before the softmax, controlling how sharply the loss separates the positive from the negatives. The whole expression is the negative log of (positive's exponentiated score) divided by (positive plus all negatives). Pushing this loss down means making q·k₊ large and every q·k₋ small.

Here is the punchline you should hold on to: this is the SAME objective as SimCLR's. Compare it to Guide 2 and the only structural change is the source of the negatives — they now come from the queue instead of from the current batch. That single swap is the whole point of MoCo: it decouples the number of negatives (the queue length) from the batch size, answering Section 1's question directly.

# f_q: query encoder (trainable)   f_k: momentum / key encoder
# queue: a buffer of K negative keys (shape C x K)
# m: momentum coefficient (e.g. 0.999)   t: temperature

f_k.params = f_q.params              # start the key encoder as a copy

for x in loader:                     # x is a small minibatch of N images
    x_q = aug(x)                     # one random view  -> query
    x_k = aug(x)                     # another random view -> key

    q = f_q.forward(x_q)            # queries  N x C  (carries gradients)
    k = f_k.forward(x_k)            # keys     N x C
    k = k.detach()                  # stop gradient: f_k is NOT trained by SGD

    # positive logits  N x 1 : each query with its own key
    l_pos = bmm(q.view(N, 1, C), k.view(N, C, 1))
    # negative logits  N x K : each query against every queued key
    l_neg = mm(q.view(N, C), queue.view(C, K))

    logits = cat([l_pos, l_neg], dim=1)   # N x (1 + K)
    labels = zeros(N)                      # the positive is always index 0
    loss = CrossEntropy(logits / t, labels)

    loss.backward()
    sgd_update(f_q.params)          # gradients update ONLY the query encoder

    # slow, smoothed update of the key encoder (the momentum rule)
    f_k.params = m * f_k.params + (1 - m) * f_q.params

    enqueue(queue, k)              # add the newest keys
    dequeue(queue)                # drop the oldest keys (keep length K)

One more practical chapter: MoCo v2. After simclr appeared, its authors showed two of SimCLR's ingredients transfer cleanly into the MoCo framework — adding a small MLP projection head on top of the encoder (instead of a single linear layer) and using stronger data augmentation. Folding these into MoCo (this upgrade is 'v2') closed most of the accuracy gap, all while keeping MoCo's small-batch, queue-based budget. The lesson: the queue/momentum machinery and SimCLR's augmentation/projection tricks are orthogonal improvements that stack.

SimCLR vs MoCo: the trade-offs

With both methods on the table, you can now reason about them as design choices rather than memorise them. They are two answers to one question — where do the negatives come from? — and each answer carries its own bill. simclr says: the negatives ARE the current batch. That is conceptually clean (no extra encoder, no extra buffer, every component trained by the same gradient) but it forces enormous batches and the high memory that comes with them. moco says: the negatives live in a queue, encoded by a slow momentum encoder. That lets a tiny batch suffice, but it pays with the added machinery of a second encoder and the consistency it must maintain.

It helps to lay the comparison on explicit axes. Memory: SimCLR is hungry (the batch must hold all negatives at once); MoCo is frugal (the queue stores cheap feature vectors, not images or activations needing gradients). Number of negatives: SimCLR is capped at roughly the batch size; MoCo is capped at the queue length, which can be far larger. Compute per step: comparable encoder cost, but SimCLR's giant batch needs many accelerators in parallel. Implementation complexity: SimCLR is simpler; MoCo adds the momentum encoder, the detach, and the enqueue/dequeue bookkeeping.

1. Memory budget: choose MoCo when accelerators are scarce — a single GPU can train it; choose SimCLR only if you already command a large parallel pod.
2. Negatives wanted: if you need tens of thousands of negatives, the queue gives them cheaply; SimCLR would need an impractically huge batch to match.
3. Simplicity vs flexibility: prefer SimCLR for a minimal, easy-to-debug pipeline; prefer MoCo when the budget constraint outweighs the extra moving parts.
4. Bottom line: with the v2 tricks shared between them, both reach broadly similar accuracy — the choice is driven by hardware, not by a quality ceiling.

What contrastive methods quietly assume

Step back and ask what is really holding these systems up. Both SimCLR and MoCo depend on negatives to supply a repulsive force. The positive term alone — 'pull the two views of an image together' — has a trivially perfect solution: map every image to the same single point. Then every positive pair is identical and the pull-together loss is zero. The only thing preventing this catastrophe is the push-apart against negatives, which forbids all embeddings from piling up in one place. Negatives are the explicit cure for representation collapse.

Say it plainly, because it sets up everything that follows: in contrastive learning, negatives are not a performance tweak — they are the structural reason the representation does not collapse. Remove them and, as written, the loss happily collapses. This is the load-bearing assumption beneath both flagships.

But negatives are also awkward, and the awkwardness is exactly what the next guide attacks. They are expensive: you need either SimCLR's huge batches or MoCo's queue and momentum encoder just to round up enough of them. And they are conceptually suspect: the positive and negative pairs are assigned by image identity, so two different photos of, say, two different dogs are labelled negatives and pushed apart — even though they share a class and arguably should be close. We are repelling things that are genuinely similar, simply because we have no labels to know better.