How an Image Becomes Tokens

The three boxes between a photo and an answer

Almost every vision-language model is built from three stages chained together: a vision encoder that looks at the image, a projector that translates what it saw into the language model's vocabulary of vectors, and the LLM itself that reasons and replies. Think of it as eye → adapter → brain. Each box is replaceable, and most of the engineering effort in multimodal models lives in the middle one.

Box one: the vision encoder

The vision encoder is a separate neural network — usually a vision transformer trained on huge image–text datasets, often a CLIP-style encoder — whose job is to turn raw pixels into a grid of rich feature vectors. It chops the image into small patches (say 16×16 pixels each), and for every patch it outputs a vector that summarizes what is there: edges, textures, objects, text. A 336×336 image might become a grid of, say, 24×24 = 576 patch vectors.

Why a pretrained encoder instead of learning vision from scratch inside the LLM? Because it is dramatically cheaper. The encoder already knows what cats, fonts, and bar charts look like from its own training; the multimodal project just has to teach the LLM how to interpret that encoder's output, not how to see from zero.

Box two: the projector — the real translator

The encoder's vectors live in its space; the LLM expects vectors in its own token-embedding space. The modality projector (also called a connector or adapter) bridges the two. In the simplest form it is just a small multi-layer perceptron that maps each patch vector to a vector of the LLM's embedding size — turning 576 patch features into 576 image tokens that look, to the LLM, exactly like word embeddings.

The projector is just a small learned map (here a two-layer MLP) sending each encoder patch vector v_i into the LLM's token-embedding space.

This little projector is doing the heavy diplomatic work: it is the piece most often trained first (with the encoder and LLM frozen) to teach the two halves to speak the same language. Some designs add a resampler here that compresses 576 patch vectors down to, say, 64 tokens, trading a little detail for a much shorter sequence — which matters because, as the next guide on cost will show, image tokens are expensive.

Box three: splicing into the sequence

Now the magic of multimodal tokenization: the projected image tokens are simply inserted into the text sequence wherever the image belongs. A prompt like "What is in <image> and why?" becomes the word tokens for "What is in," then the 576 image tokens, then the word tokens for "and why?" — one flat sequence. From here the transformer runs exactly as it does for pure text; it has no idea (and no need to know) that some tokens came from pixels.

# Conceptual pipeline (one image + a question)
patches      = split_into_patches(image)          # e.g. 576 patches
features     = vision_encoder(patches)            # 576 vision vectors
image_tokens = projector(features)                # 576 LLM-sized vectors

text_before  = embed("What is in")
text_after   = embed("and why?")
sequence     = concat(text_before, image_tokens, text_after)

answer       = llm.generate(sequence)             # plain next-token prediction

Because image tokens occupy real slots in the sequence, they eat into the context window: a single high-resolution image can cost hundreds or even thousands of tokens, the same budget you would spend on a page of text. That is why resolution handling, tiling large images, and token-compressing resamplers are such active engineering concerns.

Putting the plumbing together

1. Encoder: pixels → a grid of patch feature vectors (often a frozen CLIP-style vision transformer).
2. Projector: patch vectors → image tokens in the LLM's embedding space (the part most often trained).
3. Splice: insert image tokens into the text sequence at the right spot.
4. LLM: run ordinary next-token prediction over the mixed sequence; attention fuses image and text.