vision-language pretraining (VLP)

Vision-language pretraining teaches a model to see and read at the same time, by feeding it enormous numbers of image-text pairs scraped from the web and asking it to learn what pictures and words have to do with each other before any specific task is in sight. The hope is that after digesting a few hundred million captioned photos, the model has built a shared mental space where the word dog and a photo of a dog land near each other, so that downstream tasks like captioning or retrieval need only a light touch of fine-tuning.

Concretely a VLP system pairs a vision encoder with a text encoder or a fusion transformer and optimises one or more self-supervised objectives over paired data: image-text contrastive matching, image-conditioned masked language modelling, image-text matching as a binary classifier, and sometimes captioning. The defining property is that supervision comes from the natural co-occurrence of an image and its alt-text or caption, not from hand-drawn boxes or class labels, which is what lets the data scale to billions of pairs.

VLP is the umbrella under which dual-encoder models like CLIP, fusion-encoder models, and the encoder-decoder generators all sit. Its central difficulty is noise: web alt-text is short, generic, and frequently wrong, so a large fraction of training signal is weak, and curating or re-captioning the corpus often buys more than scaling it blindly.