cross-attention
/ kross uh-TEN-shun /
Cross-attention is attention that reaches across from one sequence to another. Where self-attention has a sequence look at itself, cross-attention lets one set of tokens consult a completely different set. The clean way to remember it: the queries come from sequence A (what I am working on), while the keys and values come from sequence B (the material I am consulting). It is how a model keeps one foot in each of two worlds.
The classic home of cross-attention is the encoder-decoder Transformer for translation. As the decoder writes the French translation, each word it produces sends a query into the encoder's stored vectors of the English source, pulling in whatever part of the original is relevant right now. That is what keeps the translation tethered to the source rather than drifting into fluent nonsense. The same trick links modalities: in a text-to-image model, the image-building tokens use cross-attention to consult the text prompt; in a captioning model, the words consult the image features.
Cross-attention is the seam where two streams of information are sewn together, and that makes it indispensable for any task that maps one thing onto another — text to image, audio to text, question to retrieved documents. Its honest limit is the same as all attention: it can only attend to material that is actually present in sequence B. If the consulted source doesn't contain the needed information, cross-attention cannot conjure it — which is one reason grounding a model in good source material matters more than the elegance of the attention itself.
In a text-to-image generator, the prompt "a red bicycle" is encoded once. As the model paints the image, the patch tokens forming the bicycle send queries into the prompt's keys/values, so the word "red" can steer the color of exactly those patches.
Queries from one stream, keys and values from another — the bridge between two sequences.
The defining feature is the split source: queries from one sequence, keys and values from another. That's the only thing separating cross-attention from self-attention — the math is identical. And it can only surface what's in the consulted sequence; it cannot invent missing information.