Flamingo gated cross-attention

Flamingo's gated cross-attention is the technique that let a frozen, pretrained language model start reading images without forgetting how to write. Rather than feeding visual tokens into the prompt, Flamingo interleaves brand-new cross-attention layers between the existing, frozen language layers, and these new layers let the text attend to a fixed-size set of visual features produced by a Perceiver-style resampler. The frozen backbone keeps all its language ability; the new layers add sight.

The crucial detail is the gate. Each inserted cross-attention and its feed-forward are multiplied by a tanh gating scalar that is initialised to zero, so at the start of training the visual layers are invisible and the model behaves exactly like the original language model. As training proceeds the gates open smoothly, easing visual information in without the destabilising shock that abruptly inserting new layers into a frozen network would cause. Attention masking also ensures each text token only attends to the images that preceded it, which is what supports interleaved image-text sequences and few-shot prompting.

This design made few-shot multimodal in-context learning work for the first time at scale, and it remains the template for adding modalities to a frozen language model when you want to preserve the backbone. The cost is many added parameters in the cross-attention layers and a fixed visual token budget set by the resampler.