CLIP and vision-language models
/ klip and VIZH-un LANG-gwij MOD-uls /
CLIP and its relatives teach a machine to relate pictures and words in a shared space. The training idea is elegant: collect hundreds of millions of images that came with text captions from the web, and train two networks — one reading images, one reading text — so that a photo and its true caption land close together, while a photo and a wrong caption land far apart. After training, the model can place any image and any sentence into the same map of meaning and measure how well they match. "A photo of a dog" lands near pictures of dogs.
This unlocks something powerful: zero-shot recognition. Because the model has linked images to open-ended language rather than a fixed list of categories, you can classify a picture by simply writing the candidate labels as sentences and asking which one the image sits closest to — no retraining needed for new categories. The same image-text alignment is the engine that lets text-to-image generators understand prompts, powers searching photos by description, and forms the visual front end of multimodal chat assistants that can discuss an image you upload.
Be honest about what this matching is and isn't. CLIP measures statistical association between visual and textual patterns scraped from the internet; it does not understand or reason about a scene. It struggles with counting, spatial relationships ("the cup left of the book"), text inside images, and anything unlike its web training data, and it inherits the internet's biases and stereotypes wholesale. It is also easy to fool with adversarial or out-of-distribution inputs. So vision-language models are remarkably flexible bridges between seeing and describing — but a high match score is a correlation, not comprehension.
Show CLIP a photo of your pet and offer three sentences: "a photo of a cat," "a photo of a dog," "a photo of a rabbit." It scores how close the image sits to each and picks the highest — say 0.31 cat, 0.02 dog, 0.01 rabbit — classifying it without ever being trained on those exact labels. Ask it instead "two cats" versus "three cats" and it often can't tell.
Classify by writing labels as sentences — flexible, but counting and spatial relations expose the limits.
Vision-language alignment is correlation, not comprehension: CLIP-style models reliably fail at counting, left/right and inside/outside relations, and reading text in images, and they inherit web-scale stereotypes. A high image-text match score means "these patterns tend to co-occur," not "the model understands the scene."