byol

BYOL (Bootstrap Your Own Latent) was the surprising result that you can learn excellent representations by pulling positives together with no negatives at all. This was thought impossible: with only an attract force and no repulsion, surely the network collapses to outputting a constant vector for every image. BYOL avoids that, and how it avoids it became one of the most studied questions in the field. The name captures the idea, the model bootstraps, it learns by trying to predict its own slightly-older representation of another view.

The architecture is asymmetric on purpose. There are two networks: an online network (encoder, projector, and a small extra predictor head) trained by gradient descent, and a target network (encoder and projector, but no predictor) whose weights are an exponential moving average of the online network. Given two augmented views, the online network sees one view and tries to predict the target network's projection of the other view; the loss is the mean-squared error between L2-normalized vectors. Critically, no gradient flows into the target network, its output is a stop-gradient target, and the loss is symmetrized by swapping which view goes to which network.

Two ingredients together prevent collapse: the predictor head on the online side (which makes the two sides asymmetric so they cannot trivially equalize) and the slowly-moving stop-gradient target (which gives the online network a stable, lagging objective to chase rather than a moving target it could collapse onto). A constant output is technically a fixed point of the loss, but it is not reached because the predictor plus EMA make the dynamics steer away from it. Early work suspected batch normalization secretly provided an implicit contrast across the batch; follow-up work showed BYOL still works with batch-independent normalization, confirming the predictor and target asymmetry are the real anti-collapse mechanism.