image captioning

Image captioning is the simplest multimodal task: look at a picture, write a sentence describing it — a brown dog catching a frisbee on a beach at sunset. It is how children learn to name the world, and it was the first bridge researchers built between vision and language. For an LLM it is almost a warm-up, the visual equivalent of saying what it sees out loud.

Captioning is exactly the training signal that aligns a vision encoder to a language model: feed the image tokens, ask the model to predict the caption one word at a time, and the gradient teaches the projector to map visual features onto the right words. Modern systems go beyond a single bland sentence to dense, paragraph-length descriptions, and these auto-generated captions are then recycled as training data for bigger multimodal models.

A fluent caption can still be wrong. Models lean on priors — beaches should have sunsets — and will describe a plausible scene rather than the actual one, smoothing over details that contradict expectation.