moco

MoCo (Momentum Contrast) is a contrastive method built around one clever idea: you do not need a huge batch to get many negatives if you keep a running memory of past negatives instead. It frames contrastive learning as looking up a dictionary, a query embedding from one view should match its single positive key (the other view of the same image) while differing from a large set of negative keys stored in a queue. The queue holds embeddings from many previous batches, so the effective number of negatives can be tens of thousands while the actual batch stays small and memory-friendly.

The subtle problem is consistency: if the encoder that produced the old keys in the queue has since changed a lot, the query and the stale keys are no longer comparable. MoCo solves this with a momentum encoder. There are two networks, a query encoder updated normally by backpropagation, and a key encoder whose weights are an exponential moving average of the query encoder's weights, updated as key = m·key + (1-m)·query with a large momentum like m = 0.999. This makes the key encoder evolve slowly and smoothly, so keys produced batches ago remain consistent enough to compare against.

MoCo decouples the number of negatives from the batch size, which was its headline practical advantage over SimCLR. MoCo v2 imported SimCLR's tricks (an MLP projection head and stronger augmentation including blur) for a large boost, and MoCo v3 adapted the approach to train Vision Transformers stably. The momentum-encoder idea also reappears, without negatives, as the target network in BYOL and the teacher in DINO, making it one of the most influential building blocks in self-supervised learning.