vision encoder

The vision encoder is the eye of a multimodal model — the part that actually looks at the pixels. Before any language happens, this network scans the image and boils it down to a compact set of feature vectors that capture there's a face here, a road there, text in the corner. It is usually a vision transformer pretrained on enormous image collections, frozen or lightly tuned, and reused across many language models.

Models like the CLIP image encoder are trained so that a picture and its caption land near each other in a shared space. That pretraining gives features already shaped for language, which is why bolting one onto an LLM works so well. The encoder outputs one vector per patch, sometimes plus a summary vector, and those go through the projector into the LLM. Swapping in a bigger or higher-resolution encoder is one of the main levers for better sight.

The encoder sets a ceiling. If it never learned to read dense text or fine medical detail, no amount of clever prompting recovers what it discarded — the language model can only reason over what the eye passed along.