Learning Without Negatives: BYOL, DINO, SwAV

Why drop negatives at all?

Negatives are the source of contrastive learning's two headaches: the appetite for huge batches or queues, and the false-negative problem where semantically identical images are wrongly repelled. So a natural question is whether we can keep only the 'pull positives together' half of the objective without collapsing. Naively, no — that is the constant-vector trap. The breakthrough of this guide's methods is that careful asymmetry makes the positives-only objective stable.

BYOL: predict the momentum target

BYOL (Bootstrap Your Own Latent) keeps two networks: an online network that is trained, and a target network that is an exponential moving average of the online one (the same momentum trick MoCo used for keys). The online network passes its embedding through an extra predictor head and is trained to match the target network's embedding of the other view. There are no negatives anywhere. The target receives no gradient — a stop-gradient seals it off.

\xi \leftarrow \tau\,\xi + (1-\tau)\,\theta, \qquad \mathcal{L}_{\theta,\xi} = \left\lVert \frac{q_\theta(z_\theta)}{\lVert q_\theta(z_\theta)\rVert} - \frac{z'_\xi}{\lVert z'_\xi\rVert} \right\rVert_2^2 = 2 - 2\cdot\frac{\langle q_\theta(z_\theta),\, z'_\xi\rangle}{\lVert q_\theta(z_\theta)\rVert\,\lVert z'_\xi\rVert}

BYOL's two moving parts: the target weights track the online weights by EMA, and the online predictor is trained to match the L2-normalized target projection.

online_z  = predictor(proj(online_enc(view1)))
target_z  = proj(target_enc(view2)).detach()      # stop-gradient!
loss = 2 - 2 * cosine(online_z, target_z)         # pull together only
# target_enc updated by EMA of online_enc, never by gradients

BYOL: an asymmetric predictor + a stop-gradient EMA target, and not a single negative.

Why doesn't BYOL collapse?

This puzzled the field for a while. The constant solution is a global optimum, yet BYOL reliably avoids it. The accepted explanation has several ingredients working together: the predictor (only on the online side) breaks the symmetry between the two branches; the stop-gradient means the target is a fixed regression goal each step, not a thing that can race to meet the online net; and the moving-average target changes slowly enough that the predictor is always chasing a near-optimal but lagging target. Remove the predictor or the EMA and BYOL collapses — both struts are load-bearing.

DINO: self-distillation with no labels

DINO reframes the same idea as knowledge distillation where the teacher is a momentum average of the student. Both output a probability distribution over a few thousand 'prototype' dimensions via softmax, and the student is trained to match the teacher's distribution on a different view. Two operations keep it from collapsing: centering (subtract a running mean from the teacher logits to stop one dimension dominating) and sharpening (a low teacher temperature to stop the distribution going uniform). Applied to a Vision Transformer, DINO produced a startling emergent property: attention maps that segment objects with no segmentation labels at all.

Notice the duality: contrastive methods fight collapse by spreading negatives out (uniformity); DINO fights the same collapse with centering+sharpening, which is uniformity and alignment enforced through the output distribution instead of through negatives.

\mathcal{L} = -\,P_t(x)^{\top}\log P_s(x), \qquad P_s = \mathrm{softmax}\!\left(\frac{g_s(x)}{\tau_s}\right), \quad P_t = \mathrm{softmax}\!\left(\frac{g_t(x)-c}{\tau_t}\right), \ \ \tau_t < \tau_s

DINO's self-distillation loss: cross-entropy from a sharpened, centered teacher softmax onto the student's softmax, with the center c removing the collapse direction.

SwAV: swap the cluster assignments

SwAV sits between contrastive and clustering. Instead of comparing features directly, it online-clusters embeddings against a small set of learned prototypes, producing a soft code for each view. Then it enforces a swapped-prediction consistency: predict view A's code from view B's features, and vice versa. Collapse (all images to one cluster) is blocked by an equipartition constraint — a Sinkhorn-normalized assignment that forces the batch to spread roughly evenly across clusters. It is contrastive in spirit but contrasts cluster codes rather than individual instances, so it scales without a giant negative bank.