visual question answering

Visual question answering, or VQA, is captioning with a purpose: instead of a free description you ask a specific question and want a specific answer. What colour is the woman's umbrella? How many people are at the table? Is the traffic light green? The model has to locate the right thing in the image and answer just that — a much sharper test of seeing than producing a vague caption.

Mechanically it is the same pipeline — image tokens plus the question tokens go into the transformer, which generates the answer — but the question steers attention toward the relevant region. VQA benchmarks have long been the standard yardstick for multimodal models, spanning everyday photos, charts, documents, and science diagrams, and instruction tuning turns plain VQA into open-ended visual chat.

VQA exposes the failure modes clearly: counting, reading small text, negation such as which person is not sitting, and questions whose honest answer is you can't tell from this image all trip models that otherwise sound authoritative.