visual instruction tuning

Visual instruction tuning is the step that turns a merely aligned vision-language model into one that follows instructions about images the way a chat assistant follows instructions about text. You assemble a dataset of image-grounded conversations, an image plus a user request plus the desired answer, and fine-tune the model to produce the answer, so afterwards it can caption, answer open questions, reason about a scene, and converse in a chosen style on command.

The trick that made this practical, introduced by LLaVA, is to generate the conversations cheaply by feeding a text-only language model the captions, object boxes, and other symbolic descriptions of an image and asking it to write rich questions and answers as if it could see. This bootstrap produces detailed-description, reasoning, and multi-turn dialogue examples at scale, which are then mixed with academic vision-language datasets reformatted as instructions. Tuning typically updates the projector and the language model, often with the vision encoder frozen.

Because the synthetic supervision is authored by a language model that never actually saw the pixels, visual instruction tuning can teach confident-sounding but ungrounded behaviour, amplifying hallucination if the captions were thin. Balancing the data mix and adding negative or grounding-focused examples is as important as scaling it.