dino

DINO (self-DIstillation with NO labels) frames self-supervision as a student copying a teacher, where both are the same network and there are no labels anywhere. A student network is trained to match the output distribution of a teacher network on different augmented views of the same image. The teacher is not a separate pretrained model; it is an exponential moving average of the student itself, so the student is, in effect, distilling knowledge from its own slowly-updated past. There are no negatives and no contrastive comparison at all.

Both networks end in a softmax that produces a probability distribution over K output dimensions (a kind of soft cluster assignment, though the dimensions have no predefined meaning). The teacher is fed the global crops while the student is fed all crops including small local ones (the multi-crop strategy from SwAV), and the student is trained by cross-entropy to reproduce the teacher's distribution. Collapse is prevented by a balanced pair of operations applied to the teacher: centering, which subtracts a running mean so no single output dimension can dominate, and sharpening, which uses a low softmax temperature so the distribution does not flatten to uniform. Centering and sharpening pull in opposite directions and together keep the output distribution healthy.

DINO's most celebrated finding is emergent behaviour in Vision Transformers: the self-attention maps of the [CLS] token spontaneously highlight object boundaries, producing usable unsupervised segmentation, and the frozen features give exceptionally strong k-nearest-neighbour classification without any fine-tuning. This suggested SSL pairs especially well with ViT architectures. The follow-up DINOv2 scaled the recipe with curated large-scale data and engineering improvements to produce general-purpose visual features that transfer across many tasks without fine-tuning.