open-vocabulary detection

Open-vocabulary detection (OVD) is object detection — drawing boxes around objects and naming them — that is not limited to a fixed list of categories fixed at training time. A classic detector trained on COCO knows exactly 80 classes and can never output 'narwhal' or 'fire hydrant cap'. An open-vocabulary detector can detect any category you name in text at inference, including categories it was never trained to detect, by aligning its visual region features with a text embedding space.

The core trick is to replace the detector's fixed classification layer (a learned weight per known class) with text embeddings of class names produced by a vision-language model such as CLIP. The detector still proposes regions and extracts a feature per region; classification then becomes: compute the similarity of each region feature to the embedding of each candidate class name, and assign the best match. Because you can embed any text, you can detect any named category. The challenge is teaching region features (local, object-level) to live in CLIP's image-level text-aligned space — addressed by knowledge distillation from CLIP (ViLD), by training on grounding data that links phrases to boxes (GLIP, MDETR, Grounding DINO), or by mixing detection and large image-caption data (Detic, OWL-ViT).

OVD is the detection counterpart of zero-shot classification and a key enabler of language-driven perception: 'find every traffic cone and stroller in this video' works without retraining a new detector for each new object. It is evaluated by splitting categories into base (seen) and novel (unseen) and reporting detection accuracy (e.g. AP) on the novel set, sometimes under the LVIS long-tailed benchmark. The same machinery powers open-vocabulary segmentation and text-prompted detection pipelines used to auto-label data.