visual grounding

Visual grounding is the task of pointing: given a natural-language expression like the man in the red jacket on the left, the model must localise the specific image region the phrase refers to, usually by predicting a bounding box or a segmentation mask. It is the bridge between naming something and showing where it is, and it is the clearest test of whether a vision-language model has truly connected its words to the pixels rather than guessing from context.

Approaches range from two-stage pipelines, which propose candidate regions and rank them by similarity to the encoded phrase, to single-stage and transformer detectors that regress the box directly from fused image-text features, to grounding-capable VLMs that emit coordinates as text tokens in their output. Open-vocabulary detectors and models trained on millions of region-phrase pairs have pushed grounding from a fixed label set toward arbitrary free-form descriptions, so the same model can localise a never-before-seen category named only in words.

Grounding matters because it makes vision-language models accountable: a grounded answer can be pointed to, which curbs hallucination and supports downstream control, robotics, and editing. The hard cases are relational and compositional expressions, the one behind the other, where the right region depends on parsing the whole scene rather than matching a single noun.