image tokenization

Image tokenization is the general act of converting a picture into a sequence of tokens that a transformer can ingest. A transformer is a sequence machine: it needs a list of vectors. Tokenization is the policy you choose for producing that list from pixels — how big the units are, how they are ordered, and crucially whether each token is a continuous vector or a symbol drawn from a fixed vocabulary, exactly like a word in text.

There are two broad families. Continuous tokenization keeps each token as a real-valued vector; the plain ViT patch embedding is the canonical example — every 16x16 patch becomes a learned D-dimensional vector and nothing is quantized. Discrete tokenization instead maps each region to one of a finite set of codebook entries, producing integer token IDs just like a text vocabulary. Discrete codes come from a learned quantizer such as VQ-VAE or VQGAN, or the dVAE used by DALL-E; the BEiT family pretrains a ViT by masking patches and predicting their discrete codes, mirroring masked-language modeling.

Why does the discrete-versus-continuous choice matter so much? Discrete tokens let you reuse the entire machinery of language modeling on images: you can model an image as a sequence of symbols with a softmax over a vocabulary, do next-token or masked-token prediction, and unify text and image generation in one autoregressive model. The cost is a quantization bottleneck — rounding every region to the nearest codebook entry throws away detail and can blur reconstructions, so codebook size and usage become central design concerns. Continuous tokens preserve information and are simpler for pure recognition, but do not slot directly into a discrete-symbol generative recipe.

Note that 'tokenization' is broader than 'patchify'. Patching is one tokenizer (a regular grid of equal tiles), but tokens can also be produced by a convolutional stem that outputs a feature-map-as-tokens, by object queries in DETR, by superpixels, or by learned merging that fuses similar regions into fewer tokens for efficiency. The choice of tokenizer fixes the granularity, the inductive bias, and the cost of everything downstream.