Taming the Data Hunger: DeiT, Distillation, and Hybrids

The data-hungry problem

In the last three guides we built a vision transformer from the ground up: we sliced an image into patches, let those patches talk through attention, and stacked the result into a deep encoder. It works beautifully — but only if you feed it enough. Here is the uncomfortable headline the original ViT paper reported: when a plain ViT and a comparable convolutional network (CNN) are both trained from scratch on a mid-sized dataset like ImageNet-1k (about 1.3 million labelled photos), the CNN usually wins. The ViT only overtakes the CNN once it has been pretrained on something enormous — like Google's internal JFT-300M, with roughly 300 million images. Same architecture, very different verdict, depending entirely on how much data you pour in.

Why? Think back to guide 1, where we met inductive bias — the built-in assumptions a model carries before it sees a single example. A CNN is born believing two things about images: that nearby pixels are related (locality), and that an object means the same thing wherever it appears (translation invariance). A plain ViT believes almost none of this. Its attention layers can connect any patch to any other patch with equal ease, so from the model's point of view a photo starts life as an unordered bag of tiles with no notion of 'near' or 'far'. Everything a CNN gets for free — locality, the idea that a cat is a cat in any corner of the frame — the ViT must discover from raw examples. Discovering a law of nature from data takes far more examples than being handed it as an assumption.

The cleanest way to hold this in your head is a scaling curve: plot accuracy on the vertical axis and dataset size (on a log scale) on the horizontal axis, and draw one line for the CNN and one for the ViT. At the left, where data is scarce, the CNN's line sits higher — its assumptions are doing the heavy lifting that data cannot yet provide. The ViT's line starts lower but climbs more steeply, because it has more freedom to exploit and fewer hand-coded assumptions holding it back. The two lines cross at some data threshold; only to the right of that crossing — in the land of hundreds of millions of images — does the ViT pull ahead and keep going. Below the threshold, the CNN's free assumptions are simply a better deal.

Inductive bias revisited: CNN's free gifts

Let's slow down and name exactly what the CNN gets for free, because each free gift is a clue to what we must add back. An inductive bias is just a prior — a built-in belief that shapes what the model finds easy to learn before any training happens. Good priors act like guardrails: they shrink the space of possibilities the model has to search, so it can find a sensible answer from far fewer examples. A CNN bakes in three priors so deeply that beginners rarely notice they are choices at all.

The first prior is locality: a convolution looks only at a small window of neighbouring pixels at a time (say 3×3), assuming that what matters most about a pixel is its immediate neighbourhood — an edge, a corner, a patch of texture. The second is weight sharing: the same small filter is slid across every position in the image. Because the identical filter is reused everywhere, a feature detected in the top-left corner is detected by the very same weights in the bottom-right. This gives translation equivariance: shift the input, and the feature map shifts the same way. Combined with pooling, it becomes rough translation invariance — a cat is recognised as a cat no matter where it sits. A ViT has neither prior by default: every patch position learns its own relationships, and nothing forces a feature learned in one corner to transfer to another.

The third gift is a multi-scale hierarchy. By alternating convolutions with pooling (which halves the resolution), a CNN sees tiny details in early layers and progressively coarser, larger structures deeper down — exactly mirroring how images are organised, from edges to textures to parts to whole objects. As the widget shows, stacking local layers makes each deep neuron's receptive field grow until it effectively sees the whole image, but it gets there gradually and locally. These three priors — locality, translation equivariance, multi-scale hierarchy — are precisely what a small-data regime craves, because they encode true facts about images that the model would otherwise have to rediscover. Here is the twist: the ViT's lack of them is its weakness (data hunger) and its strength (at huge scale it is free to learn relationships a CNN's rigid priors would forbid). That trade-off is the whole story of this guide — and it motivates the hybrid CNN-Transformer idea of putting some inductive bias back without caging the Transformer's reach.

DeiT: training ViTs on modest data

In 2021 a team at Facebook AI asked a sharp question: is the ViT's data hunger really about the architecture, or just about the training recipe? Their answer was the data-efficient image transformer, or DeiT — a ViT trained to competitive accuracy on ImageNet-1k alone, no JFT-300M required, on a single machine in a few days. DeiT rests on two pillars. The first is a far better training recipe. The second is knowledge distillation through a special new token. Together they let a Transformer learn good vision habits from a dataset you could actually download.

Pillar one is unglamorous but decisive: train it properly. Because a ViT has so few built-in assumptions, it overfits a mid-sized dataset easily, so DeiT leans hard on data augmentation and regularization to manufacture variety the data lacks. The recipe stacks RandAugment (random combinations of rotations, colour shifts, shears), Mixup and CutMix (blending two images and their labels so the model never sees the exact same example twice), random erasing, stochastic depth (randomly skipping whole layers during training), label smoothing, and the AdamW optimizer with a long warm-up and cosine decay. None of this changes the architecture; it changes the diet. The lesson is that a chunk of the apparent data hunger was really a tuning problem — give the ViT enough regularization and augmentation and it stops collapsing on ImageNet-1k.

Pillar two is the clever part. Recall the class token from guide 3 — the extra learnable token we prepend to the patch sequence, whose final state we read out as the image's summary for classification. DeiT adds a second special token alongside it, the distillation token. The class token is trained, as before, to predict the true label. The distillation token is trained to predict the output of a separate, already-trained CNN — the teacher. Picture a student Transformer sitting next to an experienced CNN tutor: the class token studies the textbook (the ground-truth labels), while the distillation token watches over the tutor's shoulder and copies how the tutor answers. Because the CNN tutor already carries locality and translation inductive bias in its bones, the student absorbs that bias indirectly — through the tutor's answers — without ever hard-coding a convolution.

DeiT's training loss: half from the class token against the truth, half from the distillation token against the teacher.

Read the equation as 'the total training loss is an equal-weight average of two cross-entropy losses, one per token.' Symbol by symbol: L is the single number we minimise. L_CE is the cross-entropy loss, which is large when a predicted probability distribution disagrees with its target. z_class and z_distill are the raw output scores (the logits) read off the class token and the distillation token respectively; softmax turns each set of logits into probabilities that sum to 1. y is the ground-truth label, and y_teacher is the CNN teacher's prediction. The two factors of one-half say we trust the textbook and the tutor equally. A tiny worked example with three classes (cat, lynx, dog): suppose the true label is cat = [0, 1, 0]. The class token's softmax is [0.2, 0.7, 0.1], so its cross-entropy is -ln(0.7) ≈ 0.357. The teacher also says cat, while the distillation token's softmax is [0.1, 0.6, 0.3], giving -ln(0.6) ≈ 0.511. The total loss is one-half × 0.357 + one-half × 0.511 ≈ 0.434. Minimising it nudges the class token toward the truth and the distillation token toward the tutor at the same time. At test time DeiT simply averages the two tokens' predictions.

Hybrid CNN-Transformer architectures

Distillation teaches a pure Transformer good habits from the outside. A hybrid CNN-Transformer takes a more direct route: bolt a small CNN onto the front. Instead of chopping the raw image into pixel patches and embedding them, you first run the image through a CNN stem — a handful of convolutional layers — to produce a feature map, and then feed those features, flattened into a sequence, as the tokens for the Transformer. This was actually one of the variants in the original ViT paper, and it is the spirit behind a great many modern vision backbones.

Why does this help? In guide 1 the standard patch embedding applied one linear projection to each raw 16×16 pixel patch — a single, shallow step with no notion of locality between patches. A CNN stem replaces that with several convolutional layers, which means the very first thing the Transformer sees is already locality-aware: each token summarises a small neighbourhood, edges and textures are already detected, and translation equivariance is already in place. In other words the stem hands the Transformer a head start of inductive bias for almost no cost. It also smooths early-layer optimization — pure ViTs are notoriously twitchy in their first few layers, and a convolutional stem stabilises the gradients so training converges more reliably.

# Hybrid ViT: a CNN stem turns an image into tokens for the Transformer.
# Instead of embedding raw 16x16 pixel patches, a few conv layers extract a
# locality-aware feature map first, then we flatten its grid into tokens.

import torch.nn as nn

class HybridStem(nn.Module):
    def __init__(self, d_model=768):
        super().__init__()
        # Each conv keeps locality + weight sharing -> free inductive bias.
        # Strided convs shrink H and W, so the feature map becomes a token grid.
        self.cnn = nn.Sequential(
            nn.Conv2d(3,    64, 3, stride=2, padding=1), nn.ReLU(),  # 224 -> 112
            nn.Conv2d(64,  128, 3, stride=2, padding=1), nn.ReLU(),  # 112 -> 56
            nn.Conv2d(128, 256, 3, stride=2, padding=1), nn.ReLU(),  # 56  -> 28
            nn.Conv2d(256, 512, 3, stride=2, padding=1), nn.ReLU(),  # 28  -> 14
            nn.Conv2d(512, d_model, 1),                              # 1x1: project to d_model
        )

    def forward(self, x):                       # x: (B, 3, 224, 224)
        f = self.cnn(x)                         # f: (B, d_model, 14, 14)
        B, C, H, W = f.shape                    # H = W = 14  ->  196 cells
        tokens = f.flatten(2).transpose(1, 2)   # (B, 196, d_model): one token per cell
        return tokens                           # hand these to the Transformer encoder

The mental picture: let the CNN do the easy local seeing — find the edges and textures that are obviously best detected with small sliding filters — and let attention handle the global reasoning about how those parts relate across the whole image. The trade-off is real, though. By committing to convolutions up front you reintroduce some of the very rigidity the ViT was designed to escape, so at extreme scale a pure ViT can edge out a hybrid. But in the data regime most practitioners actually live in, that sacrifice of flexibility buys markedly better small-data behaviour — usually a bargain.

A practical recipe: which to choose when

So which do you reach for? Four factors decide it: how much labelled data you have, your compute budget, whether a suitable pretrained model exists, and your deployment constraints (latency, memory, target hardware). The good news is that the answer is rarely 'invent something new' — it is almost always 'pick the right one of the options on the shelf.'

1. Estimate your real labelled dataset size — not how many raw images exist, but how many carry usable labels for your task.
2. Check whether a pretrained model exists for something close to your domain; if so, fine-tuning it will almost always beat training anything from scratch.
3. Weigh your compute budget: training a ViT from scratch is expensive, while fine-tuning a pretrained one or training a small hybrid is far cheaper.
4. Check deployment constraints last — required latency, memory limits, and target hardware can rule out the largest models regardless of accuracy.

Concrete rules of thumb. Little data and no pretrained weights: prefer a CNN, a hybrid, or a data-efficient image transformer with distillation — anything that injects inductive bias. A from-scratch plain ViT here is the one combination to avoid. A hybrid CNN-Transformer is a sweet spot when you want some of the Transformer's global reasoning but can only train on a modest dataset. Abundant data, or a strong pretrained ViT you can download: fine-tune the ViT and enjoy its scaling.

And here is the great equalizer, the thing that makes all of this practical for a beginner: transfer learning. You almost never train a vision transformer from scratch. Instead you download weights someone else pretrained on a huge dataset — they already paid the JFT-300M-sized 'data tax' — and you fine-tune on your few thousand images. A pretrained ViT has already learned locality and translation invariance from those hundreds of millions of examples, so on your small dataset it behaves like a model that is no longer data-hungry at all. Pretrained weights turn the ViT's biggest weakness into a non-issue for everyday work.