Pointing at Pixels: Grounding Language in the Image

From 'What' to 'Where': The Grounding Problem

The earlier guides in this track taught a machine to say what a picture is about. Captioning produces a sentence; visual question answering returns an answer. But notice what is missing: neither one ever points. If a captioning model says 'a person holding a blue mug,' it never tells you which pixels are the mug. Visual grounding is the task that closes this gap — given a natural-language phrase, it returns the exact region of the image the phrase refers to, either as a box drawn around it or as a pixel-precise mask. In one sentence: captioning and VQA answer what, grounding answers where.

Two pieces of vocabulary come back from the detection and segmentation tracks, so let us refresh them. A bounding box is the simplest 'where' — four numbers (a left, a top, a width, a height) that fence off a rectangle around an object. It is cheap and coarse: a box around a curled-up cat still includes a triangle of background in each corner. A mask is the precise alternative — a label for every single pixel saying 'this pixel belongs to the object' or 'it does not,' so the answer hugs the cat's actual outline. Grounding can output either; which one you want depends on how precise the downstream use needs to be.

Why does pointing matter so much? Consider a home robot told to 'pick up the blue mug.' Knowing a mug exists somewhere is useless; the gripper needs coordinates, so the robot must ground the phrase to a region before it can act. Augmented reality is the same — to float a label over the espresso machine, the headset must know which pixels are the machine. Accessibility tools that read a scene aloud to a blind user are far more helpful when they can say 'the exit is to your right' instead of merely 'there is an exit.' And there is a quieter, deeper reason: trust. When a system can show you where its answer came from, you can look and verify it yourself.

Referring Expressions: 'The Second Cup From the Left'

Ordinary object detection answers a category question: 'find all the cups.' A referring expression asks something harder and more human: 'find the cup I mean' — for example, 'the second cup from the left,' or 'the blue mug behind the laptop.' The task of resolving such a phrase to its single intended region is called referring-expression comprehension. The defining difficulty is that the phrase is compositional and relational: to pick the right one instance the model must resolve attributes ('blue'), ordinals ('second'), and spatial relations ('from the left,' 'behind'). Detection finds all cups and stops; grounding must use the relations to single out exactly one.

When the answer must be a pixel-precise mask rather than a box, the task is called referring expression segmentation. Same input — a sentence and an image — but the output is a stencil that follows the named object's outline exactly, not just a rectangle around it. This is the natural meeting point of language and segmentation: it is visual grounding taken to pixel resolution. A robot that needs to grasp the rim of that mug, or a photo editor that must cut out the dog on the left, wants the mask, not the box.

1. The scene: from left to right sit a red cup, a blue cup, and another red cup on a table; a laptop is behind the middle one.
2. The phrase: 'the second cup from the left.' Start by detecting every cup — that gives three candidate regions, the same set plain detection would return.
3. Resolve the noun: keep only 'cup' candidates (all three qualify). Resolve the ordinal 'second' relative to the spatial relation 'from the left': sort the three by their x-coordinate and take index two.
4. That picks the blue cup in the middle. As a sanity check, the attribute 'blue' agrees — a well-grounded model would find the relation-based and attribute-based answers consistent.

This is exactly where CLIP's known weakness bites. Guide 2 noted that CLIP-style models are strong at matching the bag of concepts in an image to the bag of words in a caption, but weak at compositional structure — they can confuse 'the cup left of the laptop' with 'the laptop left of the cup,' because the words are the same and only the relation differs. Referring expressions live or die on exactly those relations, so grounding systems usually add explicit relational reasoning on top of CLIP-like features. To know whether the region we returned is right, we need a way to score it against the human-drawn answer.

Intersection-over-Union: overlap divided by combined area.

Read this as 'how much of the two regions actually coincide.' R_pred is the region the model predicted; R_gt is the ground-truth region a human drew (the subscript gt means 'ground truth'). The symbol ∩ (intersection) is the area the two regions share — where they overlap — and ∪ (union) is the total area they cover together, counting the shared part only once. The ratio is 0 when the two regions do not touch at all (empty intersection) and 1 for a perfect match (they coincide exactly); anything in between measures partial agreement. A prediction is usually counted correct when its IoU clears a threshold, commonly 0.5. Worked example: say both the predicted and the true box have area 100, and they overlap in a region of area 80. Then the union is 100 + 100 − 80 = 120, so IoU = 80 / 120 ≈ 0.67 — above 0.5, so this prediction counts as correct. The very same formula scores boxes and pixel masks: for a mask, the areas are just counts of pixels instead of rectangle areas.

Open-Vocabulary Detection: Finding the Never-Labeled

Classic detectors have a fixed list of classes baked in. A model trained on 80 categories can find 'person,' 'car,' 'dog' — and is structurally blind to 'astronaut,' because there is no output slot for it. Open-vocabulary detection removes that ceiling by marrying detection (from the earlier track) with the CLIP-style image-text alignment from guide 2. The big idea is one swap: **replace the detector's fixed classifier head with text embeddings of class names.** Once classes are represented by the words for them rather than by fixed slots, you can detect anything you can name at test time.

Recall from the detection track that a detector has two jobs at each candidate location: say where (regress a box) and say what (classify it). In a closed-vocabulary detector, the 'what' is a fixed-size layer with one output per known class. Open-vocabulary detection keeps the 'where' machinery untouched and rewires only the 'what' so that classification becomes a matching problem between a region and a piece of text.

1. Propose regions. Run the detector's box machinery to get candidate regions (proposals) — boxes that probably contain some object, without yet saying what.
2. Embed each region. Pass each proposal through the image encoder to get a visual embedding f_r — a vector that summarizes what that patch looks like, living in CLIP's shared image-text space.
3. Embed the class names. Take your list of candidate names — 'person,' 'helmet,' 'astronaut,' anything — and run each through the text encoder to get text embeddings t_c, in the same space.
4. Match by similarity. Score each region against each name by cosine similarity and assign the region the best-matching name (above a confidence cutoff).

Region-text matching by cosine similarity, turned into a confidence.

The left equation measures alignment. f_r is the visual embedding of region proposal r; t_c is the text embedding of class name c. Their dot product f_r · t_c is large when the two vectors point the same way, and dividing by the two lengths ‖f_r‖ and ‖t_c‖ strips out magnitude so we compare pure direction — that is the cosine similarity, exactly the image-text alignment CLIP was trained to produce. It ranges from −1 (opposite) through 0 (unrelated) to +1 (perfectly aligned). The right equation turns a raw similarity into a usable confidence: σ is the sigmoid, which squashes any number into (0, 1), and τ (tau, a 'temperature') controls how sharply small score differences are amplified — a small τ makes the decision crisp, a large τ keeps it soft. (When you must choose exactly one class per region, you instead run a softmax over all the t_c.) Worked example: a region of a figure in a white suit scores cos = 0.31 against 'astronaut,' 0.22 against 'person,' and 0.05 against 'rock' — 'astronaut' wins, and after the sigmoid that 0.31 becomes a respectable confidence. Here is the punchline: swapping in different class names changes what can be detected, with no retraining at all. That is why this is genuinely zero-shot detection.

# Open-vocabulary detection: detect any class you can name.
# image_encoder / text_encoder come from a CLIP-style model (guide 2).

vocabulary = ["person", "helmet", "astronaut", "rock", "flag"]  # editable!
t = {c: l2_normalize(text_encoder(f"a photo of a {c}")) for c in vocabulary}

proposals = region_proposer(image)        # candidate boxes ("where")
detections = []
for r in proposals:
    f = l2_normalize(image_encoder(crop(image, r)))   # region embedding
    scores = {c: dot(f, t[c]) for c in vocabulary}     # cosine (vectors are unit-norm)
    best = argmax(scores)
    conf = sigmoid(scores[best] / tau)
    if conf > threshold:
        detections.append((r, best, conf))             # box, name, confidence

# To find astronauts, you just add "astronaut" to `vocabulary`.
# No labels, no gradient steps, no retraining.

Trace it through: nowhere did training data contain a box labeled 'astronaut.' The detector learned only the generic skill of proposing object-shaped regions and embedding them into CLIP's space; CLIP learned, from web-scale image-text pairs, that the word 'astronaut' lands near pictures of suited figures. At test time those two abilities meet, and a never-labeled class is found. This is the detection-shaped sibling of zero-shot image classification from guide 2: there we classified a whole image by comparing it to class names; here we do the same comparison per region, and so we get where as well as what.

Scene Graphs: Turning a Picture Into Structure

So far each phrase grounds to one region. But a picture is more than a bag of objects — it is a web of relationships. A scene graph captures that web as a graph: each object is a node, and each relationship between two objects is a directed edge labeled with a predicate. A photo of a man on a horse becomes (man) —riding→ (horse), and if he wears a hat, also (man) —wearing→ (hat). Reading the edges off gives you a compact, structured description of the whole scene rather than a flat list of things. This task is scene graph generation.

Why bother turning a picture into a graph? Because structure unlocks reasoning. A flat caption like 'a man, a horse, a hat' loses who is doing what to whom; a graph keeps it, so you can ask compositional questions ('is anyone riding?') and answer them by walking edges. It makes captioning more faithful — a model that reads the graph is far less likely to hallucinate 'the man is holding the horse' when the truth is riding. And it powers structured retrieval: 'find photos where a person is riding a horse' becomes a graph-pattern match rather than a fuzzy keyword guess.

1. Detect the objects. Run a detector to get the nodes — each a box with a category, e.g. {man, horse, hat}. (Note the dependence on the detection skills from the earlier track.)
2. Form candidate pairs. Consider ordered pairs of objects that might be related, e.g. (man, horse), (man, hat), (horse, hat).
3. Classify the predicate. For each pair, predict the relationship label — 'riding,' 'wearing,' 'next to,' or 'no relation' — using the two boxes, their relative position, and visual features between them.
4. Assemble the graph. Keep the confident predicate edges and you have the scene graph: nodes plus labeled, directed edges.

Be honest about why this is hard. First, the pairs explode: with n detected objects there are about *n × (n − 1)* ordered pairs to consider, so a busy street scene with 30 objects already means on the order of 900 candidate relationships — a combinatorial blow-up that forces models to prune aggressively. Second, predicates have a brutal long tail: a handful like 'on,' 'has,' and 'near' dominate the data, while genuinely informative ones like 'feeding' or 'reflected in' are rare. A model that only ever predicts 'on' can score deceptively well on raw accuracy while saying almost nothing, which is why scene-graph metrics emphasize recall of relationships, especially the rare ones.

Notice that every node in a scene graph is itself a grounded object — a box tied to a label — so a scene graph is really visual grounding scaled up from one phrase to the entire relational structure of an image. That is the bridge: grounding pins individual words to pixels; scene graphs pin an entire sentence's worth of structure to pixels, which is exactly the kind of verifiable, checkable scaffolding the chatty models in the next guide will need.

Promptable Grounding: Clicking, Boxing, Pointing

Everything so far steered the model with words. But words are not always the easiest handle. Picture a tangle of overlapping cables, or one specific leaf in a bush — describing the exact one in language is painful, while simply pointing at it is instant. This is the idea behind visual prompting: steering a grounding model with a visual signal instead of (or alongside) a text phrase.

Concretely, a promptable segmentation model (the SAM family — 'Segment Anything' — is the canonical example) accepts prompts like a click point ('the object under this dot'), a drawn box ('the object inside this rectangle'), or a rough scribble, and returns a precise mask for the indicated object. The interaction is real-time and iterative: click once, get a mask; if it grabbed too much, add a negative click to subtract a region; add a positive click to include more. This is interactive grounding — the human and model converge on the right region together.

Make it concrete. You open a photo editor, click once on a dog in a busy park photo, and instantly get a clean dog-shaped mask you can cut out — no typing, no waiting. A radiologist drags a box around a suspicious region and gets a pixel-tight outline to measure. A data labeler who would have spent a minute tracing an outline by hand now clicks once and corrects with a second click. In each case the prompt is a gesture on the image, and the output is a grounded region — the same 'where' we have chased all guide, reached by pointing instead of describing.

Hold these two handles side by side — a text phrase that names a region and a click that points at one — because the next guide unifies them. Multimodal LLMs accept text and visual prompts together: you can type 'mask the dog the child is hugging' and also click to disambiguate, all in one conversation. Grounding is the machinery that lets such an assistant turn either kind of prompt into an actual region on the actual pixels.

Why Grounding Is the Trust Layer

Step back and the through-line of this guide appears: grounding makes vision-language systems verifiable. A model that only says 'yes, there is a fire extinguisher' asks you to take it on faith. A model that outlines the fire extinguisher hands you the evidence — you glance at the highlighted pixels and judge for yourself whether it is right or hallucinating. That ability to show where an answer came from is exactly what turns an unfalsifiable claim into a checkable one, and so it directly attacks the deepest failure mode of these systems: confident, unverifiable hallucination.

We also picked up the toolkit for measuring grounding along the way. IoU scores a single predicted region against the ground truth — and recall that the very same formula works for boxes and for pixel masks. Recall asks what fraction of the true objects (or, for scene graphs, the true relationships) the model actually found — crucial when correct answers are rare and a lazy model could hide behind the common ones. Mask-specific metrics (mean IoU averaged over classes, or boundary-focused scores) reward hugging the true outline, not just overlapping it. Together these let us say, with numbers, whether a 'where' is right.

Be clear-eyed about what is still hard. Relational reasoning remains brittle — the same CLIP-style compositionality gap from guide 2 means 'the cup left of the laptop' is still easy to get backwards. Occlusion breaks things: when a chair is half-hidden behind a table, both the box and the mask become guesswork about the unseen part. And some failures are not the model's fault at all — language is genuinely ambiguous. 'The cup on the right' is undefined when two cups are equally rightish, and an honest system should arguably ask or offer alternatives rather than confidently pick one.

Which sets up the finale. The next guide gives language models eyes — multimodal LLMs that look at an image and chat about it fluently. Fluency is seductive and dangerous: a system that talks beautifully can also fabricate beautifully. Grounding is the component that keeps it honest, forcing the chatty assistant to point at the pixels it is talking about, and open-vocabulary detection lets it point at things no one bothered to pre-label. Read the next guide with this lens: every impressive answer is only as trustworthy as its ability to show you where.