Giving Language Models Eyes: Multimodal LLMs

The Shift: From Task Models to General Assistants

Everything earlier in this track built one specialized model per task. We trained a captioner to caption, a VQA head to answer questions, and a grounding model to point at pixels. Each was excellent at its narrow job and useless outside it. The frontier closes this gap with a different ambition: a single model that can caption, answer, reason step by step, ground its claims, hold a conversation, and follow free-form instructions it has never seen before. Instead of building a new architecture for each task, we attach eyes to a strong pretrained language brain and let it generalize. This is the multimodal large language model (MLLM), the most general member of the broader vision-language model family.

Almost every MLLM follows the same three-part recipe, and once you see it you will recognize it in every paper. First, a pretrained vision encoder plays the role of the eyes — very often the CLIP image encoder you met in guide 2, because its features already live in a space that has been aligned to language. Second, a pretrained LLM plays the role of the brain: it carries grammar, world knowledge, and the ability to reason and chat, all learned from vast amounts of text. Third, and this is where the design choices happen, a bridge connects the two so that what the eyes see can flow into the brain's reasoning. The eyes and the brain are usually borrowed off the shelf; the interesting engineering is the bridge.

Two bridging styles dominate, and the next two sections compare them head to head. The first, cross-attention bridging (the Flamingo recipe), keeps the LLM frozen and inserts new attention layers that let text tokens reach out and read the image — exactly the cross-modal attention you learned in guide 3, now wired into a giant LLM. The second, token-projection bridging (the LLaVA recipe), does something almost shockingly simple: it converts image features into a handful of vectors that look like word embeddings and just feeds them in alongside the text, letting the LLM's existing self-attention do all the fusion. Same goal, two philosophies — add machinery, or reuse the machinery you already have.

Flamingo: Slotting Images Into a Frozen LLM

The central worry when you give an LLM eyes is that retraining it on image data could damage the very language ability you are borrowing it for — a phenomenon called catastrophic forgetting, where new training overwrites old skills. Flamingo solves this with a strict rule: keep the big LLM completely frozen. Its weights never change. All of its hard-won grammar, knowledge, and reasoning are preserved untouched. Vision is then introduced only through brand-new layers that are trainable, inserted between the LLM's existing transformer blocks. The frozen backbone provides the intelligence; the small added layers learn to feed it pictures.

There is a practical problem first: a vision encoder emits a different number of feature vectors depending on the image's resolution and how many frames a video has, and an LLM wants a tidy, fixed-size input. Flamingo's answer is the Perceiver Resampler, a small attention module that takes the variable-length pile of image features and a fixed number of learned query vectors (say 64), and lets those queries attend over the features to summarize them. Whatever the input size, it always emits the same small set of visual tokens — like always writing a 64-word summary no matter whether the source was a paragraph or a whole book. This keeps the compute manageable and gives the cross-attention layers a stable, compact thing to read.

The bridge itself reuses the cross-modal attention from guide 3: the text hidden states act as queries and the visual tokens act as keys and values, so each piece of text can look up the relevant parts of the image. Crucially, Flamingo accepts interleaved image-text sequences — input like "text, image, text, image, text" in a single stream. Each text span attends to the most recent preceding image. This single design choice unlocks few-shot in-context learning with images: you can show two or three worked examples (image + desired answer) and then a fresh image, and the frozen LLM continues the pattern, all without any gradient updates — the same in-context trick text LLMs are famous for, now with pictures in the prompt.

The gated cross-attention residual update inside each new Flamingo layer.

Let us unpack this, because the small \tanh(\alpha) factor is the whole trick. y is the LLM's hidden state — the running vector representation of the text as it passes through the block. \mathrm{CrossAttn}(y_{\text{text}}, x_{\text{visual}}) is the visual cross-attention output: the text y_{\text{text}} forms queries that attend over the visual tokens x_{\text{visual}}, returning a vector that says "here is the image content relevant to this text." The update is a residual one — we add that visual contribution back onto y rather than replacing it. Now the key piece: \alpha is a single learnable scalar gate, and it is initialized to 0. Since \tanh(0)=0, at the very start the entire added term is multiplied by zero and vanishes, so y \leftarrow y + 0 = y. The new layer is a perfect no-op, and the model behaves exactly like the untouched frozen LLM. As training proceeds, gradient descent slowly moves \alpha away from zero; \tanh(\alpha) grows smoothly toward \pm 1, and vision is mixed in a little more with each step.

LLaVA: The Simple, Powerful Projection Recipe

LLaVA (Large Language and Vision Assistant) asks a wonderfully lazy question: what if we add no new attention layers at all? Flamingo's gated cross-attention works, but it is extra machinery to design, place, and train. LLaVA's bet is that the LLM's own self-attention is already a general-purpose fusion engine, so if we can just make image information look like words, the LLM will fuse it for free. This minimalism is exactly why LLaVA became the dominant open recipe: it is cheap to train (often a few hours on a handful of GPUs), trivial to implement, and easy to swap any vision encoder or any LLM into. Most open MLLMs you encounter today are LLaVA-style.

The recipe is three moves. Take the frozen CLIP image encoder and run the image through it to get a grid of feature vectors (one per image patch). Pass that grid through a small projection — just a single linear layer, or a tiny two-layer MLP in later versions — whose only job is to map each feature vector into the LLM's word-embedding space. The outputs are called visual tokens: vectors that occupy the same coordinate system as real word embeddings. Finally, simply prepend those visual tokens to the sequence of text token embeddings and feed the whole thing to the LLM as if it were one long sentence. No cross-attention is added anywhere. The LLM's self-attention then naturally lets every text token attend to every visual token — it treats image patches like extra words sitting at the front of the prompt.

The visual-token projection at the heart of LLaVA.

Reading it symbol by symbol: Z_v is the grid of image feature vectors coming out of the frozen CLIP encoder — think of it as a stack of, say, 576 vectors, one per patch, each describing what that patch looks like in CLIP's visual space. W is the trainable projection matrix (the linear version; the MLP version stacks two such maps with a nonlinearity between them), and it is the only part being learned in the alignment stage. H_v = W \cdot Z_v is the result: the same number of vectors, but each one has been moved into the LLM's word-embedding space — the exact space the LLM expects its input tokens to live in. Concretely, if CLIP gives 1024-dimensional features and the LLM uses 4096-dimensional embeddings, then W is a 4096 \times 1024 matrix and each H_v column is a 4096-vector that the LLM can read as if it were a word. Because H_v sits in that shared space, the LLM's ordinary self-attention can compute compatibility between a visual token and a text token using the very same query-key machinery it already uses for word-to-word attention. That is why no special fusion layer is needed: putting vision into the word-embedding space turns vision-language fusion into plain self-attention over a slightly longer sequence.

1. Stage 1 — Alignment (feature pre-training): freeze BOTH the CLIP encoder and the LLM, and train ONLY the projection W on a large set of image-caption pairs. The single goal is to teach W to place visual tokens where the LLM can make sense of them — it learns the 'translation' from CLIP space to word space without disturbing either pretrained network.
2. Stage 2 — Visual instruction tuning: now unfreeze the LLM (and keep training the projection) and fine-tune on (image, instruction, response) conversations so the model learns to actually be a helpful assistant — to chat, reason, and follow arbitrary visual requests. This second stage is the subject of the next section.

Visual Instruction Tuning: Teaching It to Be Helpful

After alignment, the model can fuse vision and language, but it is not yet a helpful assistant — it knows the image relates to the words, yet it has never been shown what a good, helpful, conversational answer looks like. Visual instruction tuning is the step that closes this gap. The idea is borrowed from text LLMs, where instruction tuning turned raw next-word predictors into chatbots that follow requests. Here we do the same with pictures in the loop: we fine-tune the assembled encoder-plus-LLaVA-style-LLM on many (image, instruction, response) examples, so it learns the behavior of looking, thinking, and replying helpfully — not just one fixed task, but the open-ended skill of doing whatever the user asks about an image.

1. Start from images that already have rich text annotations — human captions plus object bounding boxes with labels (the kind of grounding data from guide 4). These give a faithful, textual description of what is actually in each image.
2. Feed those captions and boxes (as TEXT, without the actual pixels) to a strong text-only LLM and prompt it to role-play a conversation: have it invent natural questions a user might ask, write detailed answers, and even produce step-by-step reasoning — all grounded in the provided description.
3. Assemble the result into (image, instruction, response) triples — often multi-turn dialogues, mixing short factual Q&A, detailed descriptions, and 'reasoning' conversations. This is how LLaVA bootstrapped a large instruction dataset cheaply, using a text LLM as a data generator.
4. Fine-tune the full multimodal model on these conversations. Now the real image IS given to the model, and it must produce the response the text-LLM wrote — learning to connect what it sees to a helpful, well-formed answer.

The autoregressive instruction-tuning loss, summed only over response tokens.

This is the standard language-model objective, adapted to condition on an image. Symbol by symbol: y_t is the t-th token of the target response (the helpful answer we want the model to produce). y_{<t} is all the response tokens that come before position t — the answer-so-far. p_{\theta}(y_t \mid \text{image}, \text{instruction}, y_{<t}) is the probability the model (with parameters \theta) assigns to the correct next token, given three things at once: the image, the user's instruction, and the partial answer already written. The expression -\log p_\theta(\cdot) is the negative log-likelihood (equivalently cross-entropy): it is small when the model confidently predicts the right token and large when it is surprised, so minimizing it pushes the model to make the true answer likely. The sum runs over t, i.e. one term per response token, and we add them up to score the whole answer. During training we use teacher forcing: instead of feeding the model its own (possibly wrong) guesses, we feed the true previous tokens y_{<t} at each step, which makes learning stable and lets every position be trained in parallel. The intuition is simply: learn to continue the helpful answer, one token at a time, while keeping the picture and the question in view.

Prompting With Pixels: Actually Using an MLLM

With an instruction-tuned MLLM in hand, how do you actually drive it? The art of getting good answers out of one is called visual prompting, and it spans a spectrum. At the simplest end is a plain text prompt with no image (the model still behaves as a normal LLM). Add a picture and you get the workhorse image-plus-text mode: "Here is a photo — what is unusual about it?" From there it gets richer: multi-image inputs let you ask the model to compare two pictures or read a sequence; pointing and set-of-marks prompting let you guide attention spatially; and chain-of-thought prompting asks the model to reason out loud, citing what it sees.

Two of these deserve a closer look because they dramatically improve reliability. Set-of-marks prompting means you literally draw numbered boxes or dots on the image before sending it — overlay "1, 2, 3" on the objects — and then phrase your question in terms of those numbers: "Is object 2 to the left of object 3?" This gives the model unambiguous visual anchors and sidesteps its notoriously shaky spatial language. Pointing is the reverse: you (or the model) reference a specific coordinate or region, tying words to an exact place. Both turn vague "somewhere in the image" reasoning into something concrete and checkable.

# Driving an MLLM: a few common visual-prompting patterns.

# 1) Image + text (the workhorse)
chat = [
    {"role": "user", "content": [
        {"type": "image", "image": photo},
        {"type": "text",  "text": "What is unusual about this scene?"},
    ]},
]

# 2) Set-of-marks: pre-annotate the image, then refer to the marks
marked = draw_numbered_boxes(photo, boxes)   # overlay 1,2,3 on objects
chat = [
    {"role": "user", "content": [
        {"type": "image", "image": marked},
        {"type": "text",  "text": "Is object 2 closer to the camera than object 3? "
                                   "Reason step by step, then answer."},
    ]},
]

# 3) Multi-image comparison
chat = [
    {"role": "user", "content": [
        {"type": "image", "image": before},
        {"type": "image", "image": after},
        {"type": "text",  "text": "What changed between the first and second image?"},
    ]},
]

# 4) Grounded chain-of-thought: force the answer to cite visual evidence
prompt = (
    "Answer the question. For each claim, name the object or region you used "
    "as evidence (e.g. 'the red sign, top-left'). Then give the final answer."
)

1. Be explicit about the task and the format you want ('answer in one word', 'list each step', 'output JSON'). MLLMs are strong instruction-followers, so format drift is usually a prompt problem, not a model limit.
2. If the question is spatial or involves counting, add visual structure first: crop to the region of interest, or use set-of-marks so the model reasons about labeled anchors instead of raw geometry.
3. Ask for chain-of-thought that cites visual evidence ('which region tells you that?'). This both improves accuracy on hard questions and makes the answer auditable.
4. For anything you must trust, close the loop with grounding: have the model return boxes for the objects it cited, then verify those boxes against the image (or a separate detector).

This last step is where the whole track snaps together. Visual grounding from guide 4 is not a separate tool you left behind — it is the verification layer for an assistant. An MLLM can fluently assert "there are three cats," but a fluent sentence is not evidence. If you instead ask it to ground each claim — "return the box for each cat you counted" — you can check the boxes, count them yourself, and catch a hallucinated cat. Grounding converts a confident-sounding answer into a checkable one, which is the difference between a demo and a system you can deploy.

The Frontier: Hallucination, Evaluation, Honesty

It would be dishonest to end the track on pure triumph, so let us be precise about what MLLMs still get wrong. They hallucinate: a model that has seen thousands of kitchen photos may confidently report a 'sink' or 'spoon' that simply is not in this image, because its language prior expects one. They are weak at precise spatial reasoning and counting — 'how many chairs?' or 'is the cup left of the plate?' trips them far more than naming objects does, because the visual tokens carry coarse, summarized location information. They are sensitive to phrasing: rewording a question, or even just offering multiple-choice letters in a different order, can flip the answer. And because they are built from a pretrained vision encoder and a pretrained LLM, they inherit the biases of both — stereotypes baked into web text and skews in what the image data depicted.

Measuring progress honestly is its own hard problem. The field leans on benchmarks — VQA accuracy, captioning scores, and large suites of probing questions — but each has pitfalls. Data leakage is rampant: if benchmark images or near-duplicates slipped into the training data, a high score can reflect memorization rather than understanding. Gameable formats are worse: a multiple-choice benchmark can sometimes be aced by a text-only model that never looks at the image, because the right option is guessable from priors and phrasing alone — a sobering check every serious evaluation should run. And aggregate accuracy hides where a model fails; a single number cannot tell you it is fine at colors but hopeless at counting. The lesson: treat benchmark numbers as weak evidence, always probe for blind-image baselines and leakage, and test on your own held-out, grounded examples.

Where is this heading, and where did we come from? The forward directions are clear: scaling the encoder, the LLM, and especially the quality and diversity of visual instruction tuning data; and extending the same three-part recipe beyond still images to video (a sequence of frames, which the Perceiver-style resampling already anticipated) and to audio, toward truly any-to-any assistants. Step back and the arc of this whole track is one clean story. We began by teaching a model to align pictures and words in a shared space (CLIP, guide 2). We learned to fuse the two modalities with cross-modal attention (guide 3). We made language ground itself in specific pixels so claims become checkable (guide 4). And finally we attached eyes to a powerful brain and instruction-tuned it into a general assistant that can look, reason, chat, and follow visual instructions (this guide). Alignment → fusion → grounding → assistants. What remains unsolved is the honest part: hallucination, reliable spatial and numeric reasoning, fair behavior across cultures, and evaluation we can actually trust. Those open problems are exactly where the next generation of work — and perhaps yours — begins.