cross-modal attention

Cross-modal attention is the mechanism that lets a model mix information from two modalities — typically letting text tokens look at image regions and image regions look at text tokens — so the two are deeply fused rather than just concatenated. Recall ordinary self-attention: each element produces a query, every element produces a key and a value, and each element's output is a weighted average of all values, weighted by how well its query matches each key. Cross-modal attention is the same machinery but with the query coming from one modality and the keys/values from the other.

Concretely, when a text token attends to image features, the text token forms a query, the image patch features form keys and values, and the resulting attention weights say 'how much should this word draw from each image region?' So the word 'umbrella' in a question can pull in the visual features of the umbrella patch. Running it the other direction lets image regions be informed by the text. This is exactly what makes tasks like VQA and visual grounding work: the language can steer where the model looks, and the visual evidence can flow into the language representation.

There are two broad architectural patterns. In dual-stream / co-attention models (ViLBERT, LXMERT), images and text have separate Transformer streams connected by cross-attention layers. In single-stream / merged-attention models (VisualBERT, UNITER), image and text tokens are concatenated into one sequence and ordinary self-attention is applied, so cross-modal mixing happens implicitly within each self-attention layer. Modern multimodal LLMs use both ideas: Flamingo inserts gated cross-attention layers into a frozen LLM so text attends to image features, while LLaVA-style models project image features into tokens and let the LLM's self-attention do the fusion.