visual grounding

Visual grounding is the task of locating the specific region of an image that a piece of language refers to. Given the phrase 'the man in the red jacket on the left' and a crowded photo, the model must return the bounding box around exactly that man, not any other person. It 'grounds' words in pixels — it builds the bridge from a linguistic reference to a concrete location, which is essential for any system that needs to act on or talk precisely about parts of a scene.

There are two closely related sub-tasks. Referring expression comprehension takes one whole expression and returns the single region it denotes (output: one box). Phrase grounding takes a sentence or caption and links each noun phrase in it to its corresponding region (output: many phrase-box pairs). Both require the model to disambiguate using attributes ('red'), categories ('jacket'), and spatial relations ('on the left', 'next to the bench') — a referring expression is, by design, the minimal description that uniquely picks out one object.

Modern grounding models fuse language and vision deeply with cross-modal attention and predict boxes either by selecting among detector proposals or by directly regressing coordinates (DETR-style). Datasets include RefCOCO, RefCOCO+ (no absolute location words allowed, forcing reliance on appearance), and RefCOCOg (longer, more natural expressions), plus Flickr30K Entities for phrase grounding. Models such as MDETR and GLIP unified detection and grounding by training on phrase-region alignment, and Grounding DINO brought this to open-vocabulary, text-prompted detection. Grounding is also a building block of multimodal LLMs that can point at what they describe (e.g. by emitting coordinate tokens).