vision-language model

A vision-language model is the most common kind of multimodal LLM: one that can see. Show it a photo and ask questions, give it a diagram and ask it to explain, screenshot an app and ask why a button is greyed out — it answers in words. It is the bridge between computer vision, which used to just label pictures, and language, which can describe, reason, and converse about what it saw.

The recipe is a strong vision encoder — often a vision transformer trained on huge image-text pairs, like the CLIP family — feeding a pretrained language model through a projector. Training has two stages: first align the vision features to the text space on millions of caption pairs, then fine-tune on instruction data so it follows requests rather than just describing. Once it can read the image, the language model's existing knowledge does most of the heavy lifting.

Their sight is shallower than it looks. They are strong at gist and named objects but weak at precise counting, exact spatial layout, fine text in cluttered scenes, and anything needing pixel-accurate measurement. Treat their visual claims as confident guesses, not measurements.