modality projector

The projector is the small adapter that lets the eye talk to the language brain. The vision encoder speaks in its own feature vectors, which don't naturally line up with the word embeddings an LLM expects. The projector translates: it takes each image feature and reshapes it into a vector that lives in the LLM's token space, so the model reads it as if it were a word it understands.

In the simplest form it is just a couple of linear layers, an MLP — a handful of parameters compared to the giant encoder and LLM. Richer designs use a small set of learned query vectors that attend over the image and compress it into a fixed number of tokens, the resampler or Q-Former idea, which controls cost. During training the encoder and LLM are often frozen and only the projector learns — a cheap way to give an existing LLM eyes.

Because so much rides on this thin bridge, it is also a bottleneck: a weak projector can waste a great encoder, and the fixed token budget it imposes caps how much visual detail ever reaches the language model.