Visual Instruction Tuning: From Captioner to Assistant

The gap between alignment and helpfulness

A model that has only seen image-caption pairs is a fluent captioner: ask it anything and it tends to describe. But users want a conversational partner — "compare these two charts", "why is this meme funny?", "transcribe the receipt and total it". Closing that gap is the job of visual instruction tuning: a supervised fine-tuning stage on image-grounded conversations, directly inheriting the recipe that turned base LLMs into assistants via instruction tuning.

Where the instruction data comes from

The bootstrap trick behind LLaVA: you usually do not have human-written multimodal dialogues, but you do have datasets with rich image annotations — captions, object boxes, region descriptions. Feed those text annotations (no pixels) to a strong text-only LLM and ask it to invent diverse question-answer conversations about the scene. The text LLM never sees the image, yet because the annotations are faithful, the generated dialogue is grounded. You then train the VLM on (image, generated conversation) pairs.

Collect images with dense annotations (captions + boxes + region text).
Prompt a text LLM to write three kinds of turns: conversational Q&A, detailed description, and complex reasoning.
Filter for faithfulness — drop turns that assert anything the annotations do not support — then fine-tune the bridged VLM on the pairs.

p_\theta(\mathbf{X}_a \mid \mathbf{X}_v, \mathbf{X}_{\text{ins}}) = \prod_{i=1}^{L} p_\theta\!\left(x_i \mid \mathbf{X}_v,\ \mathbf{X}_{\text{ins}},\ \mathbf{X}_{a,<i}\right)

Visual instruction tuning maximizes the likelihood of the answer tokens conditioned on the image and the instruction.

Interleaving images and text

Real documents are not one image and one question — they are webpages, textbooks, and chat logs where pictures and words alternate freely. Interleaved image-text modeling treats the whole thing as a single sequence with image placeholders dropped wherever they occur, and trains the model to predict the next token across both. This is what unlocks in-context multimodal ability: show two worked visual examples in the prompt, and the model picks up the pattern for a third — the multimodal analogue of few-shot prompting.

Grounding answers in regions

"Helpful" often means pointing, not just describing. Visual grounding is the ability to localize the image region a phrase refers to — "the umbrella the child on the left is holding". The modern trick is disarmingly simple: serialize a bounding box as text, e.g. `[x0,y0,x1,y1]`, and let the model emit coordinates as ordinary tokens in its answer. Now grounding, referring-expression comprehension, and even detection become special cases of next-token prediction, learned from grounded instruction data.

User: Where is the dog?  <image>
Model: The dog is on the sofa <box>[412,233,690,540]</box>.

Coordinates emitted as text tokens turn grounding into ordinary generation.

Reading documents, charts, and screenshots

A huge slice of real value is multimodal document understanding: answering questions over forms, tables, charts, and UI screenshots where layout carries meaning. The frontier approach is OCR-free — instead of running a separate text recognizer and losing spatial structure, the VLM reads pixels directly and learns to jointly reason over glyphs, position, and figures. The blocker is resolution: small fonts demand many high-res tokens, which is precisely why the resampling and tiling bridges of the last guide matter so much here.

N_{\text{tok}} = \left\lfloor \tfrac{H}{p} \right\rfloor \cdot \left\lfloor \tfrac{W}{p} \right\rfloor

The number of visual tokens grows with resolution and shrinks with patch size — too few tokens and the axis label is simply unreadable.