vision encoder and projector

In the most popular VLM recipe the two halves of the system are joined by a deliberately simple seam. A vision encoder, almost always a pretrained vision transformer, turns the image into a grid of patch embeddings; a projector then maps each of those embeddings into the language model's word-embedding space, so the image arrives as a sequence of pseudo-tokens that the language model can attend to alongside the real text tokens. The projector is the translator that makes a picture speak the language model's dialect.

LLaVA showed that this projector can be as plain as a two-layer multilayer perceptron, a striking contrast to the elaborate Q-Former or gated cross-attention alternatives. Each patch feature is sent through the same small network and the resulting tokens are simply concatenated into the prompt, so the language model handles fusion through its ordinary self-attention. Training first aligns just the projector on caption data with both backbones frozen, then unfreezes the language model for instruction tuning, with the number of image tokens scaling with input resolution.

The projector-plus-concatenation approach trades the parameter efficiency of cross-attention for simplicity and high-resolution fidelity, since every patch becomes a token the language model can read directly. Its cost is sequence length: high-resolution or multi-image inputs can flood the context with thousands of visual tokens, which is why token-merging and adaptive-resolution schemes have become important.