vision transformer

A vision transformer takes the architecture that conquered language — the transformer, which processes a sequence of word tokens by letting every token look at every other — and applies it almost unchanged to images. The trick is to stop thinking of an image as a grid of pixels and start thinking of it as a short sentence made of patches. You chop the image into a grid of small square patches (commonly 16x16 pixels), turn each patch into a single vector token, and feed the resulting sequence into a stack of standard transformer encoder blocks. There is no convolution anywhere in the core model.

Precisely: an input image of height H, width W and C colour channels is split into N = HW/P² non-overlapping patches of side P. Each patch is flattened and linearly projected to a fixed embedding dimension D (this is the patch embedding). A learnable class token is prepended, learnable position embeddings are added so the model knows where each patch sat, and the sequence of N+1 vectors passes through L identical encoder blocks, each made of multi-head self-attention and a per-token MLP with residual connections and layer normalization. The final state of the class token is sent to a small classification head. The landmark paper (Dosovitskiy et al., 2021, 'An Image is Worth 16x16 Words') showed this plain recipe matches or beats convolutional networks.

The decisive caveat is data. A convolutional network bakes in strong assumptions — locality, translation equivariance, a coarse-to-fine hierarchy — that a ViT does not have; the ViT must learn these regularities from examples. So when trained only on ImageNet-1k (about 1.3 million images) a ViT underperforms a comparable ResNet, but when pretrained on much larger corpora (ImageNet-21k with 14M images, or the private JFT-300M with 300M images) it overtakes them and keeps improving with scale. This is the headline lesson of ViT: weak built-in bias plus enough data and compute beats strong built-in bias.

ViT reframed computer vision around a single unified token-sequence backbone, and almost every modern system inherits from it: DETR-style detectors, the image and text encoders inside CLIP, masked-image self-supervision (BEiT, MAE), the conditioning backbones of diffusion models (DiT), and the promptable Segment Anything Model (SAM) all build on transformer-over-patches. Understanding the ViT is therefore the gateway to understanding contemporary vision.