data-efficient image transformer

DeiT showed that a vision transformer does not need a giant private dataset to be excellent — careful training on ImageNet-1k alone is enough. The original ViT's headline result came with a sobering footnote: it only beat convolutional networks after pretraining on JFT-300M, a 300-million-image corpus most researchers cannot access; on ImageNet-1k alone it lagged. DeiT (Touvron et al., 2021) closed that gap using exactly the same ViT architecture but a far stronger training recipe, putting competitive ViTs within reach of an ordinary lab with a few GPUs.

Its recipe has two pillars. The first is aggressive data augmentation and regularization to compensate for the transformer's weak built-in priors: RandAugment, Mixup and CutMix (which blend images and labels), random erasing, stochastic depth (randomly dropping whole layers during training), repeated augmentation, strong weight decay and label smoothing. These force the small dataset to behave like a larger, more varied one and prevent the over-parameterized ViT from memorizing.

The second pillar, and DeiT's signature contribution, is distillation through a dedicated distillation token. Alongside the usual class token, DeiT adds a second learnable token whose target is the prediction of a teacher network. Notably the best teacher is a strong convolutional network (a RegNet), so the ViT effectively learns the CNN's hard decisions — DeiT favours 'hard distillation', matching the teacher's top-1 label rather than its soft probabilities. Through this token the transformer inherits the convolutional teacher's useful inductive biases without a convolution in its own body.

The lasting significance is twofold. Practically, DeiT made ViTs trainable from scratch on standard data, democratizing transformer vision research. Conceptually, it reframed the ViT 'data hunger' problem: the issue was never that transformers fundamentally need hundreds of millions of images, but that their weak priors must be supplied somehow — by data, by augmentation, or by distilling them from a model that already has them.