momentum contrast (MoCo)

MoCo reframes contrastive learning as building and looking up a dictionary. You want lots of negative keys to contrast against, but recomputing them every step is expensive and SimCLR pays for them with huge batches. MoCo decouples the number of negatives from the batch size by keeping a running queue of recently seen embeddings — a cheap, large, and slowly refreshed bank of negatives that lives outside the current mini-batch.

The trick that makes the queue usable is the momentum encoder. The query view is encoded by the main network that receives gradients; the key views are encoded by a second network whose weights are an exponential moving average of the main network's, updated as θ_k ← m θ_k + (1−m) θ_q with m near 0.999. Because the key encoder changes only slowly, the keys sitting in the queue from several steps ago remain consistent with newly enqueued ones, which a naively-updated encoder would not guarantee. The query is trained with InfoNCE against the one positive key and the thousands of queued negatives.

MoCo v2 and v3 folded in SimCLR's lessons — an MLP projection head, stronger augmentation, and later a Vision Transformer backbone — narrowing the gap so the queue's main remaining advantage is decoupling negative count from device memory. It is the canonical answer to 'how do I get many negatives without a giant batch.'