Living Without Negatives: BYOL, SimSiam, SwAV, DINO

Representation collapse: the trap

Everything in this guide circles one enemy, so let us name it sharply. Representation collapse is the trivial, useless solution where the encoder learns to map every input image to the same constant output vector. Picture a lazy student facing a test made only of 'do these two photos show the same scene?' questions. If the answer key is always 'yes', the student who simply writes 'yes' to everything scores a perfect 100 — without looking at a single image, without learning anything. A collapsed network is exactly that student: it has discovered that it can satisfy the training objective without representing the world at all.

Why has this trap not haunted us before? Because the earlier methods in this track were built to avoid it for free. In contrastive learning (and its scaled-up cousin MoCo), the loss does two things at once: it pulls the two augmented views of one image together, but it also pushes that image away from many negatives — other, unrelated images. Those negatives are a constant outward pressure. If the network tried to collapse everything to one point, the push-apart term would explode, because every negative would sit right on top of the anchor. Repulsion makes collapse expensive, so the optimizer never goes there.

So the organizing question of this guide is simple to state and surprisingly deep: besides repulsion, what other mechanism can stop a network from collapsing to a constant? BYOL, SimSiam, SwAV and DINO are four different, ingenious answers. As we meet each one, keep a scorecard: every method must, somewhere, pay the anti-collapse tax — the only question is where it hides the payment.

BYOL: bootstrap your own latent

BYOL (Bootstrap Your Own Latent) is the result that genuinely shocked the field in 2020. It uses no negatives at all — its loss only ever says 'make these two views agree' — and yet it does not collapse, and it matches or beats contrastive methods. For a while people assumed there was a hidden bug. There was not. BYOL works because of a carefully engineered asymmetry, and understanding that asymmetry is the heart of this guide.

BYOL runs two networks. The online network (parameters θ) is the one we actually train. The target network (parameters ξ) is a slow-moving copy of the online network — its weights are an exponential moving average (EMA) of θ, never updated by gradient descent. Both networks have an encoder and a projector. The twist is the third piece: the online network has an extra little MLP called the predictor (written q_θ) bolted onto its top, and the target network has no such thing. Given two augmented views of one image, the online side encodes view 1 and then predicts what the target side will output for view 2. The target side just encodes view 2 and stays put. This is the engine that drives augmentation invariance — agreement across views — without any negatives.

The BYOL loss (top) and the EMA update of the target (bottom).

Let us read every symbol. z_θ is the online network's projection of view 1; z'_ξ is the target network's projection of view 2. q_θ(·) is the online-only predictor — the crucial asymmetry, the piece the target lacks. ⟨·,·⟩ is an inner (dot) product, and dividing by the two norms ‖·‖₂ turns the fraction into a cosine similarity between the online prediction and the target projection. sg(·) is stop-gradient: it means 'compute this value but let no gradient flow back into it', so the target is treated as a fixed goalpost for this step. The second line is the EMA update: the target parameters ξ drift toward the online parameters θ, where τ (typically about 0.99–0.999) is the decay — close to 1 means the target moves very slowly. Now the worked intuition: cosine similarity ranges from −1 to +1, so when the online prediction perfectly matches the target direction, the fraction is 1 and the loss is 2 − 2·1 = 0; when they are orthogonal, the loss is 2 − 0 = 2; when opposite, 2 − 2·(−1) = 4. The network is rewarded purely for pointing the same way as the target.

# BYOL: one optimization step (PyTorch-style pseudocode)
# f_o, g_o, q_o : online encoder, projector, PREDICTOR (all trained)
# f_t, g_t      : target encoder, projector (NO gradient; EMA of online)
# tau           : EMA decay for the target, e.g. 0.996

for x in loader:                       # x: a batch of images
    v1, v2 = aug(x), aug(x)            # two random augmented views

    # ----- online branch: encode -> project -> PREDICT -----
    z1 = q_o(g_o(f_o(v1)))            # online prediction for view 1
    z2 = q_o(g_o(f_o(v2)))           # online prediction for view 2

    # ----- target branch: encode -> project, then STOP-GRADIENT -----
    with torch.no_grad():            # sg(): no gradient enters the target
        t1 = g_t(f_t(v1))            # target projection of view 1
        t2 = g_t(f_t(v2))           # target projection of view 2

    # symmetric loss: each online view predicts the OTHER target view
    loss = D(z1, t2) + D(z2, t1)     # D(a,b) = 2 - 2 * cosine(a, b)
    loss.backward()                  # gradients update ONLY the online net
    opt.step(); opt.zero_grad()

    # target params are an EMA of online params (NOT gradient descent)
    for p_t, p_o in zip(target_params, online_params):
        p_t.data = tau * p_t.data + (1 - tau) * p_o.data

SimSiam: collapse prevention laid bare

SimSiam (Simple Siamese) is the experiment that turned BYOL's mystery into a clean answer. It asks: of BYOL's three ingredients, which are truly necessary? Then it strips the design to the bone. SimSiam throws away the EMA target entirely — the two branches are literally the same network with shared weights. It also has no negatives, just like BYOL. What it keeps is only two things: the predictor MLP and the stop-gradient. And astonishingly, it still does not collapse, and still learns strong features. This makes SimSiam the cleanest ablation in the whole literature, and the conceptual payoff of this guide.

Here is the mental model for stop-gradient. On each step, SimSiam takes one branch and freezes it — 'for the purpose of this update, you are a fixed teacher; I will fit you but not change you.' The other branch must move to match it. Then the roles swap (the loss below is symmetric, so both directions happen). Because at any instant one side is a frozen teacher, the network is never optimizing a single objective in which 'everyone becomes constant' is reachable by a smooth descent — the stop-gradient cuts the feedback loop that would otherwise drive both sides to the same trivial point together.

The symmetric SimSiam loss; D is the negative cosine similarity.

Reading the symbols: z₁ and z₂ are the projections of the two augmented views (produced by the shared encoder+projector). p₁ = h(z₁) and p₂ = h(z₂) are the outputs of the predictor MLP h applied on top — note the predictor sits on the prediction side, exactly the BYOL asymmetry, even though the weights are shared. sg(·) is stop-gradient, applied to whichever side plays the target this term. D(p, z) is the negative cosine similarity, so minimizing D means maximizing alignment. Concretely, in the first term p₁ tries to match the detached z₂, while in the second term p₂ tries to match the detached z₁; the ½ + ½ just averages the two symmetric directions. Insist on the takeaway: there is no EMA and no negatives here at all, so the only things that can be preventing collapse are the predictor and the stop-gradient.

# SimSiam: the stop-gradient is the load-bearing line
# f : encoder + projector, SHARED by both views (no EMA, no target net)
# h : PREDICTOR MLP (the only asymmetry)
for x in loader:
    v1, v2 = aug(x), aug(x)            # two random augmented views
    z1, z2 = f(v1), f(v2)             # projections -> act as 'targets'
    p1, p2 = h(z1), h(z2)            # predictions

    # D(a,b) = -cosine(a,b); .detach() IS the stop-gradient sg()
    loss = 0.5 * D(p1, z2.detach()) + 0.5 * D(p2, z1.detach())
    loss.backward(); opt.step(); opt.zero_grad()

# Try it: remove BOTH .detach() calls -> the network collapses to a
# constant within a few steps (loss -> -1). That is the whole proof.

SwAV: swap clusters instead of comparing instances

SwAV (Swapping Assignments between Views) takes a completely different escape route from negatives: online clustering. Instead of comparing one image against many other images, SwAV compares each image against a small set of learnable prototypes — think of them as cluster centers living in feature space. Each view is softly assigned to these prototypes, producing a 'code' (a soft cluster membership). The clever move is the swapped prediction: SwAV predicts the code of one view from the features of the other view. If two views really are the same object, the network should be able to guess view 1's cluster assignment just by looking at view 2 — that is the augmentation invariance signal, expressed through clusters rather than through pairwise pushes.

The swapped-prediction loss: predict each view's code from the other view's features.

Symbol by symbol: z_s and z_t are the (normalized) feature vectors of the two views, source and target. q_s and q_t are their cluster-assignment codes — soft one-hot-like vectors over the clusters. c_k is the k-th prototype (cluster center), a learnable vector; the dot product z⊤c_k measures how well a feature aligns with that prototype, and the softmax over all k (with temperature τ) turns those scores into a probability distribution p(z) across clusters. Then ℓ(z, q) is just a cross-entropy: −Σ_k q^{(k)} log p^{(k)}(z), which is small exactly when the predicted distribution p(z) puts its mass on the clusters that the code q says are correct. The swapped structure ℓ(z_t, q_s) + ℓ(z_s, q_t) means: features of view t must predict the code of view s, and vice versa. Concretely, if view s's code is essentially 'cluster #7', then the loss pushes view t's features to also light up prototype #7 under the softmax.

But wait — if we only minimized that, the network could cheat by sending every image to one cluster: then every code is 'cluster #1', predicting it is trivial, and the loss is near zero. That is collapse wearing a clustering costume. SwAV blocks it with an equipartition constraint: when it computes the codes q (not the predictions p), it requires that, across a batch, the images be spread roughly evenly over all the clusters. The codes are produced by the Sinkhorn–Knopp algorithm, a fast iterative procedure that nudges an assignment matrix until its rows and columns both sum to the target marginals — here, equal-sized clusters. So the codes are forbidden by construction from collapsing onto one prototype.

DINO: self-distillation with no labels

DINO (self-DIstillation with NO labels) pushes the teacher–student idea to its limit, and it is the bridge to the Vision Transformer era — DINO is where self-supervision and the ViT backbone meet most famously. The setup echoes BYOL but reframes it as knowledge distillation: a student network is trained to match the output distribution of a teacher network, and the teacher is an EMA of the student (so it has no labels and no separate training — it is the student's own slow average). Crucially, both networks end in a softmax over K dimensions, so 'matching' here means matching probability distributions, not just aligning vectors.

The DINO objective: the student's distribution matches the teacher's centered, sharpened distribution.

Every symbol: g_s(x) and g_t(x) are the raw outputs (logits over K dimensions) of the student and teacher networks for input x. P_s and P_t are those logits turned into probability distributions by softmax. H(P_t, P_s) = −Σ_i P_t^{(i)} log P_s^{(i)} is the cross-entropy, minimized when the student distribution P_s agrees with the teacher distribution P_t. The two anti-collapse knobs live inside the teacher's softmax. c is the center: a vector subtracted from the teacher's logits before softmax, and it is itself an EMA of recent teacher outputs. Centering stops any single one of the K dimensions from dominating — without it, the teacher could pump all its probability into one component for every image (a collapse). τ_t and τ_s are the teacher and student temperatures, with τ_t < τ_s so the teacher is sharpened: dividing by a small τ_t makes the teacher's distribution peaky and confident, giving the student a crisp target to imitate. A stop-gradient sits on the teacher, and its weights are the EMA of the student's.

Why this exact pairing? Centering and sharpening fight opposite failure modes, and that balance is the whole trick. Sharpening alone would tend to collapse: a peaky teacher pushes the student toward one dominant dimension, and the system can spiral into 'always predict component #1'. Centering alone would tend toward the other degenerate end: subtracting the running mean flattens the distribution toward uniform, which is also uninformative. Apply both together and they cancel each other's bias — sharpening keeps the output confident, centering keeps it from concentrating on one dimension — and the teacher's distribution stays varied and meaningful. That balance, not any repulsion, is exactly DINO's anti-collapse mechanism.

The unifying view: asymmetry beats collapse

Step back and the whole guide collapses (the good kind) into a single principle. Every one of these four methods removed the negatives, and every one of them had to put something else in the place where the negatives used to prevent representation collapse. That something else is always a form of asymmetry or constraint — a deliberate brokenness in the architecture or the objective that makes the constant solution unreachable. 'No negatives' never meant 'no anti-collapse mechanism'; it only moved where that mechanism lives.

1. BYOL — pays the anti-collapse tax with an EMA target plus an online-only predictor and stop-gradient: the online net chases a slow, slightly-ahead copy of itself, so it can never sit still as a constant.
2. SimSiam — strips it to the minimum: shared weights, no EMA, no negatives — only a predictor and a stop-gradient. Remove the stop-gradient and it collapses instantly, proving that piece is load-bearing.
3. SwAV — swaps to online clustering and pays the tax with an equipartition constraint (Sinkhorn–Knopp), forbidding the 'everything in one cluster' degenerate answer.
4. DINO — frames it as self-distillation and pays the tax with centering plus sharpening on the teacher, balancing the two opposite collapse modes against each other.

Notice the recurring shapes: a predictor on one side but not the other (asymmetric architecture), a stop-gradient that turns one branch into a fixed teacher (asymmetric gradient flow), an EMA that makes the teacher a slow average (asymmetric time), and a batch-level balance constraint (a global statistical rule). Any one of these is enough to break the symmetry that collapse depends on. Collapse needs both branches to be free to slide together to the same point; each method denies that freedom in a different way.

All of the methods so far — contrastive and non-contrastive alike — share one assumption: the learning signal comes from comparing whole-image views of the same picture, and the encoder is trained to make those views agree. The next guide turns that assumption on its head. Instead of comparing two views, masked image modeling hides large patches of a single image and trains the network to reconstruct the missing content — borrowing the trick that made BERT transform language. As you read it, carry this guide's lens with you: ask where (if anywhere) collapse could sneak in, and what now plays the role that asymmetry played here.