cross-modal grounding

Grounding is the act of tying a word to the exact thing it refers to in another modality — pointing, in effect. When you say the dog on the left, cross-modal grounding is the model knowing which patch of pixels that phrase picks out; in a video it is which moment, in audio which speaker. It turns a vague description into a precise link between language and the world the model perceived.

Concretely, a grounded model can output a bounding box, a segmentation mask, or a timestamp alongside its words, so the broken window comes with coordinates. It learns this from data that pairs phrases with regions or time spans, training the attention between text tokens and image or audio tokens to line up. Cross-modal attention is the underlying mechanism — text queries attending to visual keys reveal which region each word leans on.

Grounding is what separates a model that talks plausibly about an image from one you can trust to act on it — to click the right button, flag the right defect — but it remains brittle for small objects, crowded scenes, and abstract phrases that don't map to a single region.