visual question answering

Visual question answering (VQA) is the task of answering a free-form natural-language question about an image, such as 'How many people are wearing hats?' or 'What color is the umbrella?' It sits at the intersection of vision and language: the model must parse the question, look at the relevant parts of the image, and combine both to produce an answer. Unlike captioning, which says whatever is salient, VQA is directed — the question tells the model exactly what to find out.

Mechanically, VQA requires multimodal fusion: an image representation and a question representation must be brought together so the model can reason jointly. Early systems encoded the image with a CNN and the question with an LSTM, then fused them (concatenation or bilinear pooling) and classified over a fixed vocabulary of frequent answers. The influential 'bottom-up and top-down attention' approach used object detector regions as the visual units and let the question attend to them. Modern systems use cross-modal attention transformers or simply feed the image into a multimodal large language model and let it generate the answer as text, which handles open-ended answers and multi-step reasoning far better.

VQA is benchmarked on datasets such as VQAv2 (deliberately balanced so the same question has different answers across image pairs, to fight shortcut guessing), GQA (compositional questions generated from scene graphs, testing relational reasoning), and OK-VQA (questions needing outside world knowledge). A central pitfall is language priors: a model can score well by exploiting answer statistics ('what color is the banana?' → 'yellow') without truly looking at the image, which is why balanced and reasoning-focused benchmarks exist.