referring expression segmentation

Referring expression segmentation (RES) takes a natural-language phrase and an image and produces a pixel-precise mask of the single object the phrase describes — for example, given 'the slice of pizza closest to the camera', it outputs a binary mask covering exactly those pixels. It is the segmentation-level cousin of visual grounding: where grounding returns a coarse bounding box, RES returns the exact silhouette, which matters for editing, robotic manipulation, and any task where pixel boundaries count.

RES is harder than ordinary semantic segmentation in two ways. First, the target is selected by language, not by a fixed class, so the model must understand attributes and relations to pick the right instance among lookalikes (one 'slice', not all slices). Second, it must output a fine mask, requiring dense per-pixel prediction conditioned on the text. Architectures therefore fuse language features into the visual decoder — for example LAVT injects text into a hierarchical vision Transformer at multiple scales, and CLIPSeg adds a lightweight decoder on top of a frozen CLIP to turn a text (or image) prompt into a mask.

RES is evaluated on RefCOCO, RefCOCO+, and RefCOCOg with metrics such as mean Intersection-over-Union (mIoU) — the overlap between predicted and ground-truth masks divided by their union — and precision at IoU thresholds. The task connects naturally to promptable segmentation: the Segment Anything Model (SAM) produces high-quality masks from point or box prompts, so a common modern pipeline is to ground the phrase to a box (e.g. with Grounding DINO) and feed that box to SAM, yielding text-driven segmentation by composition ('Grounded-SAM').