image captioning

Image captioning is the task of automatically writing a natural-language sentence that describes what is in an image, for example turning a photo into the caption 'a brown dog catching a frisbee in a park.' It is a generation task: unlike classification, where the model picks from a fixed list of labels, captioning must produce free-form, grammatical text that is both fluent (reads naturally) and faithful (actually true of the image).

The classic recipe is an encoder-decoder. The encoder is a vision network (originally a CNN such as ResNet; today often a ViT or a CLIP image encoder) that compresses the image into feature vectors. The decoder is a language model (originally an LSTM, now a Transformer) that generates the caption one word at a time, where each predicted word is fed back in to predict the next — this is called autoregressive decoding. The landmark step was adding visual attention (the 'Show, Attend and Tell' idea): at each word, the decoder computes attention weights over image regions, so when it writes 'frisbee' it looks at the frisbee patch rather than the whole image uniformly.

Captioning is trained on datasets of image-caption pairs such as MS COCO Captions (five human captions per image) and is evaluated with metrics like BLEU and METEOR (n-gram overlap with reference captions), CIDEr (consensus weighted by TF-IDF, designed specifically for captioning), and SPICE (which compares scene-graph-like semantic tuples). Reference-free metrics such as CLIPScore measure image-caption agreement directly with CLIP, sidestepping the need for human references. Modern systems often skip task-specific training and just prompt a multimodal large language model to describe the image.