patch embedding

Patch embedding is the doorway through which an image enters a transformer. A transformer only understands a sequence of vectors of one fixed width; an image is a 2D grid of pixels. Patch embedding bridges the two: it cuts the image into a grid of small tiles and turns each tile into one vector of the required width. Think of it as photographing a mosaic, then describing each tile with a single barcode of numbers — the picture becomes a list of barcodes the transformer can read in order.

Mechanically, take an image of size H x W with C channels and a chosen patch side P. Slice it into N = (H/P) x (W/P) non-overlapping patches. Flatten each patch into a vector of length P·P·C (every pixel and channel laid end to end), then multiply by one shared learnable weight matrix that maps from P·P·C numbers down to D, the embedding dimension (plus a bias). The same matrix is applied to every patch, so the projection is shared across positions, exactly like a convolutional filter applied with stride equal to its kernel size.

That equivalence is worth stating plainly: a patch embedding is identical to a single 2D convolution whose kernel size and stride both equal P. Most implementations literally use one Conv2d layer with kernel=stride=P for speed. This is the one and only place where the raw 2D structure of the image is consumed; after this step the model sees only an unordered set of D-dimensional tokens, which is why position information must be added separately.

Two design knobs matter. The patch size P sets the sequence length and therefore the compute: halving P quadruples the number of tokens N and, because attention costs grow with N², roughly multiplies attention cost by sixteen — smaller patches give finer detail at steep cost. And because the projection is linear and shared, it cannot model anything inside a patch beyond a linear summary; some hybrid models replace it with a few convolutions (a 'convolutional stem') to inject a little locality and stabilize early training.