JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

The Promise of Self-Supervised Learning

Why we learn representations without labels, what a pretext task is, and how to judge a representation you cannot directly see.

The label bottleneck

Supervised learning works, but labels are expensive, slow, and politically loaded — someone has to decide what each example means. Meanwhile the world is awash in unlabeled images, text, audio, and video. Self-supervised learning (SSL) is the bet that we can extract most of what a representation needs to know directly from the structure of the data itself, holding labels in reserve for a thin final layer. The goal is not to predict a label but to learn a representation: a mapping from raw input to a vector that makes downstream tasks easy.

Self-supervised learning lives inside the unsupervised paradigm — it manufactures its own supervisory signal from unlabeled data.

Three learning paradigms side by side — supervised, unsupervised, and reinforcement learning.

Pretext tasks: a short history

A pretext task is a problem we do not actually care about, solved only so that the representation learned along the way is useful. Early computer-vision pretext tasks were charmingly literal: predict the relative position of two image patches, solve a jigsaw puzzle, colorize a grayscale photo, or guess the rotation applied to an image. Each forces the network to understand objects, parts, and context — but each also leaves the network free to cheat by latching onto low-level shortcuts (chromatic aberration at patch borders was a notorious one).

The decisive modern pretext task is instance discrimination: treat every image as its own class. With N training images you imagine an N-way classification problem where each image (and its augmented variants) is a unique category. Instance discrimination turns out to be far harder to cheat than jigsaw puzzles, and almost every contrastive method is a variation on it.

Two ingredients: a pretext task and an evaluation protocol

Because the pretext loss is not the thing we care about, an SSL paper lives or dies by its evaluation protocol. The standard is linear probing: freeze the pretrained backbone, throw away the projection head, and fit a single linear classifier on top using the real labels. A linear probe can only succeed if the representation has already done the hard work of organizing classes linearly — so high linear-probe accuracy is strong evidence that the features are good, not that some fancy head rescued them.

  1. Pretrain the backbone on unlabeled data with your chosen SSL objective.
  2. Freeze every backbone weight; discard the projection / prediction heads used only for SSL.
  3. On a labeled set, train ONLY a linear layer on the frozen features and report accuracy.
  4. Cross-check with k-NN accuracy, fine-tuning, and low-shot probes to be sure the gain is not protocol-specific.
Linear probing freezes the backbone and fits only a linear decision boundary on top — if the frozen features are good, a straight line already separates the classes.

A straight decision boundary separating two classes of points.

The spectre that haunts every method: collapse

There is a trivial way to make any 'make two views of the same image agree' objective go to zero: map every input to the same constant vector. The views agree perfectly and the representation is perfectly useless. This is collapse, and it is the central antagonist of this whole track. A subtler version, dimensional collapse, leaves the embeddings varying but confined to a low-dimensional subspace — many of the directions you paid for carry no information.

\mathcal{L}_{\text{agree}} = \big\lVert f(x) - f(x') \big\rVert_2^2 \;\xrightarrow{\;f \equiv c\;}\; 0

The collapse failure mode: if the network maps every input to one constant vector c, any 'make the two views agree' loss trivially drops to zero — yet the representation is useless.

A map of the ladder ahead

We will climb in four moves. First, contrastive methods that prevent collapse with explicit negative examples. Second, distillation methods like BYOL and DINO that drop negatives entirely and lean on asymmetry. Third, redundancy-reduction methods (Barlow Twins, VICReg) that prevent collapse by regularizing the embedding's statistics directly. Fourth, masked and predictive methods (MAE, VQ-VAE, JEPA) that reconstruct or predict hidden content. The same representation can later anchor transfer to detection, segmentation, retrieval, and multimodal models.