flamingo

Flamingo, from DeepMind (2022), is a vision-language model built for few-shot learning: you give it a handful of image-and-text examples in the prompt and it learns the task on the fly, the way large language models do in-context learning, but now with images interleaved among the words. Its big idea was to bolt vision onto an already-trained large language model so the LLM's reasoning and few-shot abilities transfer to multimodal inputs, without retraining the whole thing.

Architecturally, Flamingo keeps the LLM frozen (originally Chinchilla) and keeps the vision encoder frozen, then inserts new trainable components between them. A Perceiver Resampler converts a variable number of image features into a small fixed number of visual tokens (taming the cost of long visual sequences). Then gated cross-attention layers are interleaved into the frozen LLM: at these layers text tokens attend to the visual tokens. Crucially the gate starts at zero (a tanh gating initialized so the new layer contributes nothing at first), so adding vision does not disrupt the pretrained language model, and the model smoothly learns to use the image signal during training.

Flamingo natively handles interleaved sequences of images and text — multiple images, each followed by text, in one prompt — which is exactly what enables multimodal few-shot prompting (show two captioned examples, then a new image, and it captions it). It demonstrated strong few-shot results on captioning, VQA, and visual dialogue without task-specific fine-tuning, and its frozen-LLM-plus-gated-cross-attention design became a template for later interleaved multimodal models (e.g. the open OpenFlamingo and IDEFICS reproductions).