visual instruction tuning

Visual instruction tuning is the fine-tuning step that turns a raw image-plus-language model into a helpful conversational assistant that follows instructions about images. The analogy is text-only instruction tuning, which taught base LLMs to answer questions and follow commands rather than merely continue text. Visual instruction tuning does the same for multimodal models: it trains them on examples of the form (image, instruction, desired response) so they learn to describe, answer, reason, compare, and converse about images instead of just emitting a flat caption.

The bottleneck is data: high-quality image-grounded conversations are scarce. The LLaVA insight was to generate them cheaply by leveraging a strong text-only LLM. Take an image that already has rich annotations (captions, object boxes, region descriptions from datasets like COCO), feed those text annotations to GPT-4, and prompt it to invent diverse dialogues — detailed descriptions, multi-turn Q&A, and complex reasoning — as if it were looking at the image. The result is a large synthetic instruction-following dataset, which is then used to fine-tune the multimodal model. Later work added human-annotated and academic-task data, GPT-4V-generated data, and harder reasoning and OCR-heavy examples.

Why it matters: instruction tuning is what makes a multimodal model usable. A model that has only been aligned (CLIP-style) or only pretrained on captions can match or describe, but it will not naturally answer 'why is this funny?' or 'extract the table into JSON' or hold a multi-turn conversation. Visual instruction tuning is also where behaviors like format-following, refusal, and grounding are shaped — and where data quality directly controls hallucination, since teaching the model to state only what is visible reduces confident but false descriptions.