visual prompting

Visual prompting means steering a model with cues placed in or alongside the image, rather than only with words. Just as you can prompt a language model by phrasing text, you can prompt a vision model by pointing: clicking a point, drawing a box, scribbling, or marking regions tells the model where to focus or what to operate on. The clearest example is the Segment Anything Model (SAM), which is promptable — you give it a point or a box and it returns the mask of the object there — turning interactive selection into a first-class input.

The term covers two fairly different ideas. The first is inference-time spatial prompting: overlaying visual cues (points, boxes, circles, or numbered marks) that a model already understands. A striking finding is that drawing a red circle around an object can make CLIP attend to it, and 'Set-of-Mark' prompting — overlaying numbered segmentation masks on an image and referring to objects by number — markedly improves the visual grounding of multimodal LLMs like GPT-4V. These cues require no training; they exploit a model's existing sensitivity to salient marks.

The second idea is parameter-level visual prompt tuning (VPT): instead of fine-tuning a whole vision Transformer for a new task, you freeze it and prepend a small number of learnable 'prompt' tokens (or learn an image-space perturbation) that are optimized on the task. This is the vision analog of prompt tuning in NLP — a parameter-efficient adaptation method that adjusts a tiny set of inputs while leaving the large backbone untouched. Both senses share the spirit of prompting: change the input, not the weights, to control behavior.