blip

BLIP (Bootstrapping Language-Image Pre-training), from Salesforce (2022), is a vision-language model designed to be good at both understanding tasks (retrieval, VQA) and generation tasks (captioning) in one framework — whereas CLIP only does matching and not text generation. Its architecture, the Multimodal mixture of Encoder-Decoder (MED), can operate in three modes from shared weights: a unimodal encoder (CLIP-like contrastive matching), an image-grounded text encoder (for image-text matching), and an image-grounded text decoder (for generating captions).

BLIP's signature contribution is CapFilt, a data-bootstrapping method that cleans noisy web data with the model itself. Web captions are often inaccurate, so BLIP fine-tunes a captioner to generate new synthetic captions for web images and a filter to discard captions (web or synthetic) that don't actually match the image. Training on this bootstrapped, higher-quality data noticeably improves results — hence 'bootstrapping' in the name: the model improves its own training set.

BLIP-2 (2023) reframed the problem around efficiency by keeping a powerful frozen image encoder and a frozen large language model and training only a small bridge between them called the Q-Former (Querying Transformer). The Q-Former holds a fixed set of learnable query tokens that extract the most language-relevant visual features and feed them to the LLM, so BLIP-2 reaches strong captioning and VQA at a fraction of the trainable parameters. This 'frozen encoders + small learned connector' pattern became hugely influential for later multimodal LLMs.