simsiam

SimSiam is a stripped-down sibling of BYOL that asks how minimal a non-contrastive method can be and still avoid collapse. It is a Siamese network, two branches sharing the same encoder, fed two augmented views of an image, trained to make the two outputs agree. There are no negative pairs, no large batches, no memory queue, and crucially no momentum encoder at all. The entire anti-collapse mechanism reduces to one operation: a stop-gradient.

In detail, both views pass through the same encoder and projector. One branch additionally passes through a small predictor head, and its output is matched (by negative cosine similarity) to the other branch's projection. The other branch's projection is given a stop-gradient, meaning it is treated as a fixed target with no gradient flowing back through it. The loss is symmetrized so each view plays both roles. That is the whole method, and the central experimental finding is that removing the stop-gradient causes immediate, total collapse to a constant, while keeping it yields strong representations.

SimSiam's contribution is conceptual clarity: it isolates the stop-gradient as the essential ingredient and shows the momentum encoder of BYOL and the negatives of SimCLR, while helpful, are not strictly necessary to prevent collapse. The authors propose an interpretation as an alternating-optimization (expectation-maximization-like) procedure, where the stop-gradient corresponds to fixing one set of variables while updating the other, which is why optimizing both sides simultaneously without it degenerates. It remains a favourite reference point for understanding why non-contrastive self-supervision works at all.