vision-language models (VLMs)

A vision-language model is a single system you can show a picture and talk to: it takes images and text together as input and produces text out, so you can ask it to describe a photo, read a chart, answer a question about a diagram, or reason step by step about what it sees. The modern incarnation is essentially a large language model that has been given eyes, retaining its fluency and world knowledge while gaining the ability to ground its words in pixels.

The dominant recipe wires a pretrained vision encoder to a pretrained language model through a connector: the encoder turns the image into a grid of patch features, a projector or querying module maps those features into the language model's token space, and the language model treats them as if they were extra tokens in the prompt. Training proceeds in stages, first aligning the connector on caption data with the backbones frozen, then visual instruction tuning the assembly on image-grounded conversations so it follows arbitrary user requests.

VLMs inherit both the strengths and the failure modes of their language backbone: they hallucinate objects that are not present, struggle with precise counting and spatial relations, and are limited by the resolution at which the encoder sees the image. Evaluating them honestly requires probing for grounding rather than rewarding plausible-sounding text.