Learning Without Labels: The Self-Supervised Idea

Why labels are the bottleneck

Modern image recognition was built on a simple but expensive idea: show a neural network millions of pictures, each stamped with a human-written label — cat, dog, fire truck — and let it learn the mapping. The most famous such dataset, ImageNet, holds about 1.3 million images sorted into 1,000 categories. Producing it took years of effort and an army of human annotators clicking on image after image. If you wanted a fresh labelled dataset at that scale today, you would be looking at millions of dollars and many months of work before a single model could even start training.

Now contrast that with what the internet offers for free. Every single day, people upload billions of photos to social media, photo libraries, and the open web — utterly without tidy labels. This is the painful asymmetry that motivates this whole track: labelled data is scarce and expensive, while raw, unlabelled images are almost infinite and essentially free. The dream is obvious: what if a model could learn from the unlabelled billions instead of the labelled millions?

Here is a clue that the dream is realistic — think about a human child. Long before any adult teaches them the word 'dog', a baby has already discovered an enormous amount about how the visual world is put together: that objects move as connected wholes, that faces have two eyes above a nose, that light usually comes from above, that things look smaller when they are far away. Nobody hands the child an answer key. The structure of the world itself is the teacher. The child watches, quietly predicts what will happen next, and is corrected by reality when it is surprised.

Put plainly, self-supervised learning is the art of getting the data to grade its own homework. We will spend this guide turning that one-line idea into something precise: how to manufacture such a task, what we actually want the model to walk away with, and how to measure whether it succeeded. By the end you will have the full vocabulary needed to read the rest of the track.

Supervised vs self-supervised: where the signal comes from

Let's make the difference precise, because it is the hinge the whole field turns on. In ordinary supervised learning, every training example is a pair: an input image and a target that a person wrote down. You show the network a photo, it guesses 'dog', you compare that guess to the human label 'cat', and the mismatch — the loss — is the signal that nudges the network's weights. The learning signal is, quite literally, human effort baked into every single example.

Self-supervised learning keeps that exact machinery but changes one thing: where the target comes from. Instead of paying a person to write the answer, we manufacture a task whose correct answer is already contained in the data. The classic move is to hide or transform part of an image and ask the network to recover what we hid. Erase a square from a photo and ask the model to paint it back. Convert a colour picture to grayscale and ask it to predict the colours. Rotate an image and ask by how much. In every case the original, untouched image is the answer key — and it cost nothing.

# Supervised: the target comes from a human annotator
for image, human_label in labeled_dataset:
    prediction = network(image)
    loss = cross_entropy(prediction, human_label)   # answer written by a person
    loss.backward(); optimizer.step()

# Self-supervised: the target is manufactured from the data itself
for image in unlabeled_dataset:
    view, target = make_pretext_task(image)   # e.g. rotate image; target = rotation angle
    prediction = network(view)
    loss = cross_entropy(prediction, target)        # answer written by the data
    loss.backward(); optimizer.step()

The made-up puzzle we invent — 'predict the rotation', 'fill in the hole' — has a name: a pretext task. The word pretext is chosen to be a little dismissive on purpose. It signals that the task is a cover story, an excuse to make the network stare hard at images. We do not actually care whether the model becomes a world-class rotation detector or a champion at reassembling shuffled patches.

Why not? Because the pretext task is only a means to an end. What we are really after is everything the network is forced to learn along the way in order to solve the puzzle — a rich internal understanding of objects, parts, textures, and layout. The puzzle is scaffolding; the understanding is the prize. We will make that prize precise in the very next section, and in Section 4 we will see exactly how a clever pretext task forces real understanding to emerge.

What is a representation, really?

Everything in this track revolves around one object, so let's define it carefully. A representation (also called an embedding) is a compact list of numbers — a vector — that stands in for an image. The component that produces it is the encoder: a neural network that swallows a messy, high-dimensional image (a 224×224 colour photo is over 150,000 raw pixel numbers) and squeezes it down to a short vector, perhaps 2,048 numbers long. The study of how to learn good encoders is exactly representation learning.

The encoder map: an image x becomes an embedding vector z.

Read this as 'the representation z is what the encoder f, with parameters θ, computes from the image x.' Symbol by symbol: x is the input image — literally a grid of pixel values. f_θ is the encoder, a neural network; the subscript θ (theta) stands for all of its learnable weights, the millions of numbers adjusted during training. z is the output, the embedding vector — say a single point sitting in 2,048-dimensional space. There is no algebra to solve here; the equation is just a precise name tag for each piece. The whole game of SSL is to find weights θ so that this map turns images into useful points.

What makes one z 'useful' and another useless? Think of summarising a 400-page novel into five keywords. A lazy summary ('book, pages, words, ink, cover') technically describes it but tells you nothing about what it means. A good summary ('orphan, wizardry, friendship, boarding-school, courage') captures the essence, so that two novels with similar summaries really are similar stories. A good representation does the same for images: it arranges them in vector space so that semantically similar images land close together and different ones land far apart. If every dog photo clusters in one region and every truck in another, then telling dogs from trucks becomes trivially easy for anything built on top.

Pretext tasks: clever puzzles with free answers

Early self-supervised vision ran on hand-crafted pretext tasks, and four of them became classics. Rotation: rotate each image by 0°, 90°, 180°, or 270° and ask the network which of the four it was. Jigsaw: cut the image into a grid of patches, shuffle them, and ask the network to recover the original arrangement. Colourisation: strip an image to grayscale and ask the network to predict its original colours. Inpainting: delete a region and ask the network to fill the hole with plausible content. In all four, the answer is free — you created the puzzle, so you already hold the solution.

Walk through rotation slowly, because it shows the magic. Suppose I hand the network an image rotated by 90°, and it must answer 'rotated 90°'. How could it possibly know? Only by understanding the content. It has to know that skies belong at the top and ground at the bottom, that faces are normally upright, that trees grow upward, that printed text runs left-to-right. A network that has internalised all of that can read the orientation straight off the scene; a network that sees only meaningless colour blobs cannot. So the harmless-looking question 'how is this rotated?' secretly forces the model to learn object structure — purely as a side effect of solving the puzzle.

This reveals the golden rule of pretext design: a good pretext task is one that cannot be solved by a cheap, low-level shortcut. If the network can crack the puzzle without understanding content, it will — and you will get garbage features. Jigsaw is a cautionary tale: if you leave the patches touching, the model can reassemble them by matching edge textures or the tiny colour fringes at the borders — like cheating at a real jigsaw by feeling the cut shapes instead of looking at the picture. The fix is to leave gaps between patches so the shortcut disappears and only genuine understanding survives.

Measuring success: the linear evaluation protocol

A fair objection: if pretraining never touched a single label, how can we possibly know whether the representation it learned is any good? We need a yardstick. The standard one is the linear evaluation protocol, and it is beautifully simple — a clean, four-step recipe that almost every paper in this track reports.

1. Pretrain the encoder on unlabelled data using your self-supervised pretext task.
2. Freeze the encoder — lock every weight θ so it can never change again.
3. On top of the frozen encoder, train just one new linear classifier, using a labelled dataset.
4. Measure that classifier's accuracy on held-out test images. That single number is your representation's score.

The linear probe: one trainable layer on top of a frozen representation z.

The linear probe is exactly this one equation, so let's name every part. z is the frozen representation the pretrained encoder produced for an image — it is fixed, the probe cannot change it. W is the weight matrix of the single trainable linear layer (the 'probe'); b is its bias vector; together `Wz + b` just computes a raw score for each class as a weighted sum of z's components plus an offset. softmax then squashes those raw scores into positive numbers that add up to one — a probability distribution. ŷ (y-hat) is that predicted distribution over the classes, e.g. '88% dog, 9% cat, 3% truck'. Now the punchline, and the reason the probe is deliberately linear: a linear layer can only carve flat boundaries through the embedding space — it has almost no power of its own. So if a frozen encoder plus this feeble layer can still classify accurately, the encoder must have already done the hard work, arranging the images so the classes are linearly separable. That is precisely what we mean by 'a good representation'.

The road ahead: families of SSL

You now hold every concept needed to make sense of the rest of this track. Modern self-supervised vision splits into three big families, and the remaining four guides take them one at a time. Here is the map so the whole arc feels intentional rather than like a parade of tricks.

Contrastive methods (Guides 2 and 3) make two different views of the same image — say two random crops — and train the encoder to pull those two together in embedding space while pushing other images away. The 'pushing away' needs lots of comparison examples called negatives, and a surprising amount of engineering goes into supplying enough of them. That is the story of contrastive learning and the methods that scaled it up to enormous numbers of negatives.

Negative-free methods (Guide 4) ask an awkward question: can we pull matching views together without any negatives to push apart? The danger is collapse — the encoder cheating by mapping every image to the same constant vector, which trivially makes all views 'agree' while learning nothing at all. This family borrows tricks from distillation and clustering to actively prevent collapse, and includes the well-known methods BYOL, SimSiam, SwAV, and DINO.

Masked image modeling (Guide 5) returns to the oldest idea in this very guide — hide part of the data and predict it — but at full modern strength. It masks out large chunks of an image and trains the network to reconstruct the missing content, directly echoing how BERT learned language by filling in blanked-out words. This is masked image modeling, the approach that carried the humble 'predict the missing piece' recipe to its most powerful form in vision.