segment anything model

The Segment Anything Model (SAM), released by Meta AI in 2023, is a foundation model for segmentation: instead of being trained for one fixed set of classes, it learns the general skill of "outline the thing I'm pointing at" and can then segment objects it was never explicitly taught, with no fine-tuning. You give it a prompt — a click, a box, a rough mask, or (to a limited degree) text — and it returns a mask. This promptable, zero-shot behavior is the same paradigm shift that large language models brought to text, applied to pixels.

Architecturally SAM has three parts engineered around a speed asymmetry. A heavy image encoder (a Vision Transformer pretrained with masked autoencoding) runs once per image to produce a rich embedding — this is the expensive step. A lightweight prompt encoder turns clicks, boxes, or masks into tokens. A fast mask decoder then fuses image embedding and prompt to emit a mask in milliseconds, so a user can click around interactively in real time after the one-time image encoding. Because a single click is ambiguous (click a shirt — do you mean the shirt, the person, or the whole group?), SAM is trained to output three masks at different granularities plus a predicted IoU score to rank them.

What made SAM possible was its data engine: a model-assisted annotation loop that bootstrapped the SA-1B dataset of about 11 million images and over 1 billion masks, far larger than any prior segmentation set. SAM 2 (2024) extends the same promptable idea to video, adding a streaming memory so a mask prompted on one frame is tracked through time. SAM is best understood as a general-purpose mask proposer and an annotation accelerator; it produces class-agnostic masks (it outlines but does not name), so pairing it with a classifier or an open-vocabulary model like CLIP is common when category labels are needed.