Letting Vision and Language Talk: Cross-Modal Attention and Fusion

Why Matching Isn't Enough

In the last guide we met CLIP, a model that learns to push the embedding of a picture and the embedding of its caption close together. The payoff is beautiful: CLIP turns an entire image into ONE vector and an entire sentence into ONE vector, and then it just measures the angle between them. That is exactly what you want for retrieval — give me the text 'a dog on a skateboard' and I can rank a million images by how close each one's single vector sits to the text's single vector. Fast, simple, scalable.

But now ask a harder question: 'What colour is the woman's umbrella?' This is visual question answering (VQA), and CLIP simply cannot do it. Why? Because CLIP only ever produced one summary vector for the whole image — it never zoomed in on the specific umbrella patch that the question cares about. A single global vector blends the sky, the street, the woman, and the umbrella all together. The information about 'umbrella = red' is in there somewhere, smeared across the vector, but there is no mechanism for the WORD 'umbrella' to go and fetch the visual features of that one region.

This is the deeper version of the compositionality weakness we flagged in guide 2: because CLIP collapses everything into one vector, it struggles to keep track of WHICH attribute belongs to WHICH object. 'A red umbrella and a blue coat' and 'a blue umbrella and a red coat' can land at almost the same point. Matching two summaries can tell you a picture and a sentence are roughly about the same scene, but it cannot reason about the relationships between the parts. To answer a question or to caption precisely, the text needs to be able to look INTO the image and query its individual parts.

A Quick Recap: Attention as Soft Lookup

Before we generalize attention across modalities, let's make sure the original idea is rock solid — the next section is just a small twist on it. The cleanest way to think about attention is as a SOFT LOOKUP, like a search engine that never returns just one result. You type a query; the search engine compares it against the keys (titles, tags) of every document; instead of handing you the single best match, it gives you a blended answer that is mostly the best matches and a little of the others.

Attention gives every item three roles, each a learned vector. The QUERY (Q) is 'what I am looking for'. The KEY (K) is 'what I advertise myself as' — the label other items use to find me. The VALUE (V) is 'the actual content I hand over once I'm selected'. Keys and values come in pairs: the key decides HOW MUCH attention an item gets, and the value is WHAT that item contributes. Separating the two is powerful — an item can be easy to find (a distinctive key) while carrying rich content (a different value).

Scaled dot-product attention.

Let's unpack every piece. Q is the matrix whose rows are the query vectors (one per item that is asking); K stacks the key vectors and V the value vectors (one row per item being looked at). The product QK^{\top} is the heart: each entry is a dot product between one query and one key, i.e. a relevance SCORE — how well does this query match this key? A bigger dot product means 'more relevant'. We divide by \sqrt{d_k}, where d_k is the dimension of each key vector; without this, in high dimensions the dot products grow large, push the softmax into a near-one-hot spike, and kill the gradients. Dividing by \sqrt{d_k} keeps the scores at a sane scale so the softmax stays smooth. The \operatorname{softmax} then turns each ROW of scores into positive weights that sum to 1 — a probability distribution over 'how much to listen to each item'. Finally, multiplying by V returns, for each query, a weighted average of the value vectors: mostly the values of the items it scored highly, with a sprinkle of the rest.

A tiny worked example makes it click. Suppose d_k=4, so we divide by \sqrt{4}=2. A query q=[1,0,1,0] looks at two items with keys k_1=[1,0,1,0] (a great match) and k_2=[0,1,0,1] (no overlap). The raw scores are q\cdot k_1=2 and q\cdot k_2=0; after scaling they become 1 and 0. Then \operatorname{softmax}([1,0])=[\,e^1/(e^1+e^0),\,e^0/(e^1+e^0)\,]\approx[0.73,\,0.27]. So the output is 0.73\,v_1+0.27\,v_2 — overwhelmingly the value of the matching item, but never a hard 100/0 choice. That 'soft' blend is exactly why attention is differentiable and trainable.

One last recap piece: real Transformers don't run attention once, they run MULTI-HEAD attention — several attention 'experts' in parallel. Each head gets its own learned projections of Q, K, V into a smaller subspace, so one head might specialise in colour relationships, another in spatial layout, another in counting. Their outputs are concatenated and mixed by one more linear layer. Think of it as asking the same lookup question through several different lenses at once, then combining the perspectives. Keep this picture handy — the cross-modal version we build next is literally this same machinery, with one change in where the vectors come from.

Cross-Modal Attention: Words Looking at Pixels

Here is the whole guide in one sentence: in cross-modal attention, the queries come from one modality while the keys and values come from the OTHER. Concretely, we let the text tokens be the queries and the image patches be the keys and values. Now every word can issue a query that searches across the image patches and pulls back a weighted blend of their visual features. The word is no longer stuck inside the sentence — it reaches across the bridge into the picture.

Cross-attention: queries are projections of text tokens; keys and values are projections of image patches.

Read it slowly, because the only new idea lives in the subscripts. T is the matrix of TEXT token features (one row per word); P is the matrix of IMAGE patch features (one row per patch, exactly the patch embeddings a Vision Transformer produces). W_Q, W_K, W_V are learned projection matrices — the same kind we used in self-attention. The twist: Q=W_Q T is built from the TEXT, while BOTH K=W_K P and V=W_V P are built from the IMAGE. So when we compute QK^{\top}, every word's query is scored against every image patch's key, giving a 'which patches matter for this word' weight per word; and multiplying by V returns, for each word, a weighted blend of image features. The formula is identical to before — it is still \operatorname{softmax}(QK^{\top}/\sqrt{d_k})V — only the SOURCE of the vectors changed.

Now walk the umbrella example. The sentence 'what colour is the woman's umbrella?' is tokenised, and the word 'umbrella' becomes a query vector q_{\text{umbrella}}=W_Q\,t_{\text{umbrella}}. The image is cut into patches, each with a key and value. The dot products q_{\text{umbrella}}\cdot k_{\text{patch}} are large for the handful of patches that actually contain the umbrella and small everywhere else, so after softmax the weight piles onto those umbrella patches. The returned value is therefore dominated by the umbrella's visual features — including its colour. The word 'umbrella' has effectively asked the image 'where are you, and what do you look like?' and gotten an answer. This is exactly the fine-grained lookup that lets a model do visual question answering.

There is a lovely bonus: those softmax weights ARE an attention map, and you can paint them back onto the image as a heatmap. Pick the word 'umbrella', read off its weights over the patches, and the bright region shows you precisely where the model looked when it processed that word. This is one of the few moments in deep learning where the internals are genuinely interpretable — you can literally see a word's gaze. (Read these maps with care, though: high attention means 'the model weighted this region', not a guaranteed causal explanation. Guide 4 will turn this gaze into a real ability to POINT at regions.)

Architectures: Dual, Fusion, and Both

With self-attention and cross-attention in hand, you can now read the whole map of vision-language architectures. The first family is the DUAL-ENCODER, and you already know it: CLIP. Two separate towers — an image encoder and a text encoder — each produce their own embedding with NO communication between them. Because the towers are independent, you can precompute and store every image embedding offline; at query time you only encode the text and do a cheap dot product. That is why CLIP scales to billions of images. The price is that the towers never interact at the level of parts, so fine-grained reasoning (VQA, attribute binding) is out of reach.

The second family is the FUSION-ENCODER. Here we insert cross-modal attention layers so that text and image features genuinely mix: words attend to patches (and often patches attend back to words). This is far better at VQA, captioning, and any task that needs reasoning about relationships between parts — it is exactly the interaction CLIP lacked. The catch is speed. Because the answer depends on BOTH inputs flowing through the cross-attention layers together, you cannot precompute an image's representation independently of the text. Every single image-text PAIR must be pushed through the model. Scoring one text against a million images now means a million full forward passes instead of a million dot products.

So which do you pick? In practice, both — in a pipeline called RETRIEVE-THEN-RERANK. First use a fast dual-encoder to cheaply shortlist, say, the top 50 candidate images (or captions) out of millions using precomputed embeddings. Then run the slow but accurate fusion model on only those 50 pairs to rerank them with full cross-attention reasoning. You get the dual-encoder's scalability AND the fusion-encoder's precision, paying the expensive computation only on a tiny, already-good shortlist. This pattern shows up everywhere a vision-language model must serve real traffic.

Where exactly do the cross-attention layers sit? Inside a standard Transformer block on the text side, we keep the usual self-attention sub-layer (so words can talk to each other), and then add a SECOND attention sub-layer — the cross-attention one — where the text queries attend to the image keys/values. After that comes the usual feed-forward network, and as always every sub-layer has a residual connection and normalization. Stack a few of these blocks and the text representation gets repeatedly refined by the image: self-attention organises the sentence, cross-attention injects the visual evidence, layer after layer.

# One fusion Transformer block on the text stream (pseudocode).
# text:  (num_words,   d)   -- token features (the queries)
# image: (num_patches, d)  -- patch features from a vision encoder (keys/values)

def fusion_block(text, image):
    # 1) Self-attention: words talk to each other (Q,K,V all from text)
    text = text + self_attention(q=text, k=text, v=text)
    text = layer_norm(text)

    # 2) Cross-attention: words look at the image
    #    Q comes from TEXT, but K and V come from the IMAGE patches
    text = text + cross_attention(q=text, k=image, v=image)
    text = layer_norm(text)

    # 3) Position-wise feed-forward, as in any Transformer
    text = text + feed_forward(text)
    text = layer_norm(text)
    return text  # text now carries fused visual evidence

BLIP: Unifying Understanding and Generation

Let's ground all of this in one concrete, modern design: BLIP. BLIP is a fusion model built to do something CLIP cannot — it both UNDERSTANDS images (retrieval, VQA) AND GENERATES text about them (captioning). It pulls this off by training one set of shared image-and-text encoders against three complementary objectives at once. Think of the three objectives as three different exams that force the same model to learn three different skills from the same image-caption data.

1. Image-Text Contrastive (ITC): the CLIP-style objective. Pull matching image and caption embeddings together and push mismatched ones apart, so the two modalities live in a shared, aligned space. This makes the dual-encoder path fast at retrieval and gives the harder objectives a well-organised starting point.
2. Image-Text Matching (ITM): a binary 'do these actually go together?' classifier. Crucially, this head uses cross-attention so it can do a fine-grained check — not just 'roughly the same topic' but 'does every detail line up?'. It is trained with hard negatives (mismatches that look almost right), forcing the model to notice subtle binding errors like a red umbrella mislabelled blue.
3. Image-conditioned Language Modeling (LM): generate the caption word by word, conditioned on the image via cross-attention. This is what gives BLIP a mouth — the ability to WRITE. The same generative head powers both image captioning and visual question answering (a question is just a conditioning prompt, and the answer is generated text).

Why all three? Because each covers a gap the others leave. ITC aligns the spaces cheaply but only coarsely; ITM adds fine-grained verification via cross-attention; LM adds the ability to actually produce language, not just score it. Together they make ONE model that can retrieve, judge a match, AND write a caption — understanding and generation in a single backbone.

There is a second BLIP idea worth knowing: CapFilt, short for Caption-and-Filter. Web image-text data is enormous but noisy — alt-text is often junk ('IMG_2043.jpg', 'click here'). CapFilt cleans it with the model's own skills. A CAPTIONER (the LM head) generates a fresh synthetic caption for each web image, and a FILTER (the ITM head) then throws out any caption — original OR synthetic — that doesn't truly match the image. The surviving captions form a cleaner training set, which trains a better model, which can caption and filter even better. Cleaner data lifts quality because the model stops wasting capacity memorising noise and learns from descriptions that actually describe the picture.

What Fusion Buys You, and What It Costs

Let's tie the thread together. Plain matching, as in CLIP, is fast and scalable but blind to the parts — it compares two summaries and cannot reason about which attribute goes with which object. Cross-modal attention fixes this by letting every word reach into the image and gather exactly the patches it needs. That single capability unlocks [[visual-question-answering|VQA], grounded captioning, and genuine multi-part reasoning — the abilities a model like BLIP is built on.

But fusion is not free. Because the score of an image-text pair depends on both inputs flowing through the cross-attention layers together, you can no longer precompute and cache per-image vectors the way CLIP does — every pair needs a full forward pass. That is the entire economic reason the retrieve-then-rerank pattern exists: lean on the cheap dual-encoder to narrow the field, then spend the expensive fusion compute only on a small, promising shortlist. Whenever you design a multimodal system, this speed-versus-precision trade-off is the first knob to think about.

There is also an honest limit to keep in mind. Even with cross-attention, a fusion model still answers in WORDS. When you ask 'what colour is the umbrella?' it says 'red' — it may have looked at the right patches internally, but its OUTPUT is text, not a box drawn around the umbrella. If you need the model to actually point at the exact region — to localise — that is a different capability. Guide 4, 'Pointing at Pixels: Grounding Language in the Image', is exactly about turning that internal gaze into an explicit pointer.

And there is a second frontier. The models in this guide are specialists — fine-tuned to retrieve, match, caption, or answer. They are not yet general assistants you can give arbitrary instructions to ('describe this chart, then write a tweet about it'). Bolting a powerful language model onto a vision encoder, so visual features flow into an instruction-following LLM, is the leap to multimodal LLMs — the subject of guide 5, 'Giving Language Models Eyes'. Cross-modal attention is the bridge that makes both of those next steps possible; you now understand the mechanism every one of them is built on.