Why freeze, why bridge
Training a vision encoder and a large language model each costs a fortune. The dominant recipe for vision-language models therefore reuses both: take a pretrained vision encoder (often the CLIP image tower from the last guide) and a pretrained LLM, freeze most of their weights, and train only a small bridge that turns image features into something the LLM can read. This is parameter-efficient by construction — you train millions of bridge parameters, not billions of backbone parameters.
Every design choice below is really an answer to one question: *how do visual features get into the language model's attention?* They split into two camps — input-space bridges that turn images into pseudo-tokens the LLM reads like words, and layer-wise bridges that inject vision through new cross-attention inside the LLM.
The simplest bridge: a projector
The minimal viable bridge is a vision-encoder projector: a tiny MLP (sometimes a single linear layer) that maps each image-patch embedding into the LLM's token space. The projected patch vectors are simply prepended to the text tokens as if they were words. The LLM's own self-attention then fuses them with the prompt for free. This is the LLaVA recipe — and its astonishing lesson is that a two-layer MLP plus good instruction data is enough to build a capable assistant.
The projector is just a small MLP mapping each patch embedding e_i to a pseudo-token h_i in the LLM's input space.
patch_feats = vision_encoder(image) # (P, d_vis), frozen vis_tokens = mlp_projector(patch_feats) # (P, d_llm), trained seq = concat(vis_tokens, embed(text)) # treat patches as tokens out = llm(seq) # self-attention fuses both
Resampling with learned queries: Perceiver and Q-Former
Instead of passing all P patches, resample them to a fixed K (say 32 or 64). The Perceiver idea is to keep a small array of learned latent queries and let them cross-attend over the (large) set of image features. The latents — not the image — carry forward, so cost decouples from input size. Flamingo's Perceiver Resampler does exactly this to turn a variable number of patches (or video frames) into a fixed handful of visual tokens.
Diagram of attention as a query–key–value lookup weighted by softmax.
Q-Former (from BLIP-2) refines the same trick into a two-stage bridge. A set of learnable query tokens passes through a small transformer that cross-attends to the frozen image features; the queries are also trained against text, so the bridge is taught to extract language-relevant visual evidence — not just any compression. The resulting query outputs become the visual prefix for the LLM. The whole Q-Former is tiny relative to either backbone, and both backbones stay frozen.
Injecting vision layer-by-layer: gated cross-attention
The other camp keeps the text tokens untouched at the input and instead inserts new cross-modal attention layers between the LLM's existing blocks. This is Flamingo's gated cross-attention: every few layers, the text hidden states attend over the resampled visual tokens. The catch — and the elegance — is a tanh gate initialized at zero. At the start of training the gate is closed, so the augmented model is bit-for-bit identical to the original frozen LLM; the gate opens gradually as the new layers learn, which keeps the pretrained language abilities from collapsing.
Flamingo-style gated cross-attention adds a tanh gate initialized at zero, so the new visual path contributes nothing at step 0.
Choosing a bridge
- Want the simplest path and have an instruction dataset? Use an MLP projector (LLaVA-style). It is shockingly strong and easy to debug.
- Token budget tight, or feeding video / many images? Resample with a Perceiver or Q-Former to a fixed small count.
- Must keep the LLM's text behavior pristine and interleave many images? Gated cross-attention (Flamingo) preserves the frozen LLM exactly at init.
A bridge tells the model how to look. It does not yet tell it how to be helpful — to answer questions, follow instructions, and ground its claims. For that we need data and a tuning stage, which is the next guide.