vision-language model

A vision-language model (VLM) is a single model that takes in both pictures and words and reasons about them together, instead of treating sight and language as two separate problems. Imagine showing a friend a photo and asking 'what is the person on the left doing?' — your friend must look at the image, understand your sentence, connect the word 'left' to a location in the picture, and answer in words. A VLM is software trained to do exactly this kind of joint seeing-and-talking.

Concretely, a VLM has at least two parts: a vision encoder that turns an image into a list of numerical vectors (for example a Vision Transformer, ViT, that splits the image into patches and produces one vector per patch), and a language component that processes text tokens (a Transformer). The two are connected so that information flows across the 'modalities' — image and text are two modalities. The connection can be a contrastive alignment (as in CLIP, where image and text are pushed into a shared embedding space) or a deep fusion (where text tokens attend to image features through cross-modal attention), and the output can be a similarity score, a class, a bounding box, or generated text.

VLMs are the umbrella under which most modern multimodal tasks live: image captioning, visual question answering, image-text retrieval, zero-shot classification, visual grounding, and multimodal chat are all things a sufficiently general VLM can do. The field has moved from task-specific models (one network per task) toward broad, pretrained foundation models (CLIP, BLIP, Flamingo, LLaVA, and the multimodal large language models behind systems like GPT-4V and Gemini) that are pretrained on huge image-text corpora and then adapted with little or no task-specific data.