visual instruction following

Visual instruction following is the skill that turns a model that can merely describe a picture into one that does what you ask about it. Crop out the receipt and total the amounts, explain this circuit to a beginner, is anyone not wearing a helmet — the instruction is in words, the subject is in the image, and the model must combine both to act. It is the multimodal version of the instruction-tuning that made chat LLMs helpful.

After the base vision-language alignment, the model is fine-tuned on datasets of image, instruction, and good-answer triples — often generated by prompting a strong text LLM with image captions and bounding boxes, then writing question-answer pairs. This teaches it to attend to the relevant part of the image, follow the verb in your request, and format the reply, rather than dumping a generic caption.

The instruction data shapes the behaviour, so blind spots are inherited: if the tuning set never asked it to count or to read tiny labels, it will follow such instructions confidently and wrongly.