From Pixels to Patches: How an Image Becomes a Sequence

Why invent a new way to see?

If you arrived here from the earlier vision tracks, you already met the convolutional neural network (CNN). Its core move is the sliding filter: a tiny window of learnable weights (say 3×3 pixels) slides across the image, and at each position it looks at just that small neighbourhood and produces one number. Stack thousands of these little detectors into a layer, then stack many layers, and the network slowly learns edges, then textures, then shapes, then objects. Each neuron only ever sees a small patch of the input — its receptive field — and the network grows its understanding by going deeper.

Here is the honest limitation. Because each filter is local, a CNN can only relate two far-apart corners of an image after the signal has passed through many layers. Information about the top-left of a photo has to be relayed, layer by layer, before it can ever meet information about the bottom-right. The network does eventually 'see globally,' but only slowly, through depth. Global context is something a CNN earns, not something it has from the start.

The Vision Transformer takes the step-back view. Its promise is simple and bold: let every part of the image look at every other part immediately, in the very first layer, with no waiting for depth to relay the message. Remarkably, it does this with the exact same machinery — the Transformer — that reshaped natural-language processing, where a model reads a whole sentence and lets every word attend to every other word at once. The single leap in this guide is realising we can turn that same architecture on images. But to feed an image into a machine built for sentences, we first need to make an image look like a sentence. That is the trick of the next section.

The big trick: cut the image into patches

A sentence is a sequence of words. An image is a grid of pixels. To make one look like the other, the Vision Transformer does something almost childishly simple: it chops the image into a regular grid of fixed-size square patches — small tiles, like the pieces of a mosaic, or like cutting a comic book page into its panels. This step of breaking a continuous image into discrete pieces is called image tokenization, because each tile is about to become a 'token,' the visual analogue of a word.

Let us make it concrete with the standard setup. Take an image that is 224 pixels tall and 224 pixels wide, and choose a patch size of 16×16 pixels. Along the height we fit 224 ÷ 16 = 14 patches; along the width, another 14. That gives a 14×14 grid, which is 14 × 14 = 196 patches. So this single image becomes a sequence of 196 tiles — a 196-'word' sentence. Each tile is one token, and the whole point of the rest of the architecture is to let these 196 tokens talk to each other.

The number of patches a tokenized image produces.

Let us read this equation symbol by symbol. N is the number of patches — the length of our sequence. H is the image height in pixels and W is its width in pixels; here both are 224. P is the patch side length in pixels — the size of one square tile — here 16. The division H/P counts how many patches fit down the height, and W/P how many fit across the width; multiply them and you get the total grid count. Plugging in: N = (224/16)·(224/16) = 14·14 = 196, exactly the number we counted by hand. Note this is integer division — the tiles must tile the image perfectly, so the side length must be divisible by P (224 is divisible by 16; an awkward size like 225 would not be, and is usually resized or padded first).

Turning a patch into a token: patch embedding

We have tiles, but a tile is still a little square of raw pixels — the Transformer wants each token to be a single vector of numbers, the same way a language model wants each word to be a vector. The step that converts a pixel tile into such a vector is patch embedding, and it is the bridge that completes image tokenization. It happens in two small moves: first flatten the patch's pixels into one long list of numbers, then project that list through a learned matrix to a fixed-size vector.

Think of patch embedding as giving each tile a name tag — a compact vector of numbers that summarises what is in that tile (a slice of fur, a bit of sky, the corner of a wheel). This is exactly the visual version of a word-embedding, where the word 'cat' becomes a vector that captures its meaning. After this step, every one of our 196 tiles is a name tag, and the image is a genuine sequence of vectors — a sentence the Transformer can read.

Projecting one flattened patch into the embedding space.

Symbol by symbol: x_p is the flattened patch — a single row vector of length P·P·C, where C is the number of colour channels (3 for red, green, blue). For our 16×16 patch that is 16·16·3 = 768 raw numbers (the brightness of every colour at every pixel in the tile). E is the learned projection matrix; its shape is (P·P·C)×D, that is 768×D, and it maps the raw 768-long list to a vector of the chosen embedding dimension D. The classic ViT picks D = 768, so a 768-input list is turned into a 768-dimensional name tag (D need not equal the input size — that 768=768 here is just a popular coincidence). b is an optional bias vector of length D, a small fixed offset added to every embedding. Multiplying x_p by E and adding b gives z, the final token vector.

import numpy as np

P, C, D = 16, 3, 768          # patch side, colour channels, embedding dim
N = (224 // P) * (224 // P)   # 14 * 14 = 196 patches

# patches[i] is one 16x16x3 tile; flatten each to length P*P*C = 768
patches = np.random.rand(N, P, P, C)
x = patches.reshape(N, P * P * C)   # shape: (196, 768)

# E is LEARNED during training, not hand-designed
E = np.random.randn(P * P * C, D)   # shape: (768, 768)
b = np.zeros(D)                     # bias, length 768

z = x @ E + b                       # shape: (196, 768) -> 196 token vectors
print(z.shape)                      # (196, 768)

The class token and the whole pipeline

There is one more token we add by hand — and it carries no pixels at all. Before the 196 patch tokens, we prepend a single extra learnable vector called the class token (often written [CLS]). It is not a piece of the image; it is a blank slot whose values are learned during training. Its job is to gather information from all the patches as they talk to one another, and by the end to stand in for the whole image when we make the final prediction.

The analogy: imagine a class representative or note-taker sitting in a room full of patches. The note-taker has nothing to say at first, but listens to everyone, absorbs the gist of the whole conversation, and at the end delivers one summary. We then read only the note-taker's final vector and hand it to a small classifier to decide 'cat' or 'dog.' Why a dedicated token instead of, say, averaging all patches? Because a learnable token can flexibly decide what to gather and from whom, rather than being forced to weight every patch equally.

1. Patchify: cut the input image into a grid of fixed-size square patches (224×224 with P=16 → 196 patches).
2. Embed: flatten each patch and project it through the learned matrix E into a D-dimensional token vector.
3. Prepend the class token: add the extra learnable [CLS] vector at the front, giving 197 tokens in total.
4. Add positions: add a position signal to each token so the model knows where each patch sat in the original grid.
5. Encode: pass all tokens through a stack of identical Transformer encoder blocks, where patches repeatedly attend to one another.
6. Read out & classify: take the final class-token vector and feed it to a small classification head to predict the label.

That is the whole Vision Transformer in one breath. Notice how much of it is just the bookkeeping we built in this guide — patchify, embed, prepend, add positions. The two boxes we have deliberately left closed are where the real magic lives: how patches attend to one another (the heart of attention, in guide 2) and how an encoder block stacks attention with the rest of its parts and repeats with depth (the encoder, in guide 3). For now, the one idea to carry forward is the leap itself: an image, chopped into patches and tagged, is just a sequence — a sentence the Transformer already knows how to read.

What we gave up, what we gained

Every architecture makes assumptions about the world before it sees any data, and those built-in assumptions are called its inductive bias. A CNN is packed with helpful ones. Locality: it assumes pixels that are near each other are related, which is why its filters only ever look at small neighbourhoods. Translation equivariance: it assumes an object means the same thing wherever it appears, so a cat-detector that works in the top-left will work in the bottom-right too, because the very same filter slides everywhere. These assumptions are true of natural images, so they act like training wheels — they steer the network toward sensible solutions before it has seen much data.

The Vision Transformer throws most of these training wheels away. By letting every patch attend to every other patch from layer one, it makes almost no built-in assumption that nearby patches matter more, or that position should be handled in a fixed way (it even has to add position signals by hand, because pure attention treats the patches as an unordered set). This is the trade we made: in exchange for instant global reach and great flexibility, the ViT must learn locality and translation structure from data rather than getting them for free.

The clean way to picture it: a CNN is a specialist for images, born already believing the right things about pixels, so it learns efficiently from modest data — but those beliefs also box it in. A ViT is a generalist, arriving with very few prior beliefs, free to discover whatever structure the data reveals — but with nothing to lean on, it needs to see far more examples before its raw flexibility turns into real skill. More freedom and instant global reach on one side; a serious hunger for data on the other.