image tokens

A language model thinks in tokens — little chunks it reads one after another. To let it see, an image has to be chopped into a comparable stream. The picture is sliced into a grid of small square patches, say 14 by 14 pixels each, and every patch becomes one vector — an image token. A single photo might turn into a few hundred of these, lined up like a sentence the model can read alongside your words.

Each patch is flattened and passed through the vision encoder, which produces a feature vector; a projector then maps it into the model's embedding space so it lives next to word vectors. Position information is added so the model knows which patch sat where. From the transformer's point of view these are just more entries in the sequence — it attends over them exactly as it does over text.

Image tokens are expensive. High-resolution images can blow up into thousands of tokens, eating context and slowing inference, so systems downscale, tile, or pool patches to keep the count manageable — trading visual detail for speed.