vision transformer (ViT)
/ VIZH-un trans-FOR-mer (V-I-T) /
A vision transformer applies the transformer — the same architecture that powers modern language models — to images. The clever trick is to treat a picture like a sentence. It chops the image into a grid of small square patches (say 16×16 pixels each), lines them up in order, and treats each patch as a "word." Then it uses self-attention, the transformer's core mechanism, to let every patch look at every other patch and weigh how relevant they are to each other. A patch of fur can attend to a patch of ear far across the image to jointly conclude "cat."
This was a notable departure. For nearly a decade, convolutional networks (CNNs) dominated vision; they scan an image with small local filters and have a built-in assumption that nearby pixels matter most. ViT throws out that assumption and lets attention learn long-range relationships directly. The 2020 paper that introduced it was bluntly titled to make the point that, with enough data, you don't need convolutions at all to do excellent image recognition.
The crucial caveat is that very last clause: with enough data. Because ViT has no built-in sense of locality, it must learn that structure from scratch, which means it is data-hungry — on small datasets a plain ViT can lose to a CNN. It shines when pretrained on enormous image collections and then fine-tuned. In practice, the lines have blurred: many modern systems mix transformer and convolutional ideas, so "ViT versus CNN" is less a war than a menu, chosen to fit the data and compute you have.
A ViT cuts a 224×224 photo into 196 patches of 16×16 pixels. To recognize a flamingo, the patch showing its curved neck can directly attend to the patch showing its pink legs — even though they sit far apart in the image — instead of relaying the signal pixel by pixel as a convolution would.
Image as a sequence of patches; attention links distant patches directly. The cost is a hunger for data.
ViT's edge over CNNs mostly shows up at large scale; on modest datasets a CNN's built-in locality often wins. "Transformers replaced CNNs in vision" overstates it — many state-of-the-art systems are hybrids, and the right choice depends on your data and compute budget.