Segment Anything: Interactive and Foundation Models

The promptable paradigm

For four guides we have built segmenters the old way: choose a fixed list of classes (road, person, car…), gather labelled images, and train a network to color every pixel with one of those labels. That works beautifully — until someone asks for a class you never trained on, or an object that does not fit any category at all. The whole pipeline is locked to the label list you committed to on day one. This last guide is about escaping that lock. We are going to make a single model that segments on demand: you give it an image plus a small hint of what you mean, and it returns exactly that mask.

That hint is called a prompt. A prompt can be a single click on the thing you want, a box drawn loosely around it, a rough scribbled mask, or — in the newest systems — a piece of text like 'the dog'. The model's job is no longer 'assign each pixel to one of K classes'; it is 'given this prompt, return the mask the user intends'. This is interactive segmentation taken to its logical extreme, and it changes the contract completely: there is no fixed category list, and segmentation becomes a conversation rather than a one-shot prediction.

If this feels familiar, it should. Back in guide 1 we met GrabCut: you dragged a box around an object and a graph-cut algorithm figured out the foreground inside it. A box is a prompt. GrabCut is the spiritual ancestor of everything in this guide — the same idea of 'tell me roughly where, I'll give you the precise mask' — but powered by hand-built color models and graph optimization instead of a deep network. What changes now is the engine: we keep GrabCut's promptable spirit and replace its hand-crafted machinery with a model that has seen a billion masks.

Here is the plan for the guide. First we revisit classic interactive segmentation — the click-and-scribble methods that came between GrabCut and today — so you can feel exactly what limited them. Then we open up the Segment Anything Model (SAM): its architecture, how it was trained, and what 'anything' really means. Finally we get practical: how to evaluate these foundation segmenters with the very same overlap metrics from earlier guides, and how to compose SAM with other models to get labelled, useful results. By the end you will have a single thread running from classical image segmentation all the way to the open-world frontier.

Classic interactive segmentation

Before deep learning, interactive segmentation already worked, just laboriously. The core idea is seeds: the user marks a few pixels they know are foreground (say, green strokes) and a few they know are background (red strokes), and the algorithm propagates those labels to every other pixel. GrabCut does this with graph cuts — it builds a graph where each pixel is a node, neighbouring pixels are linked by edges whose weight is high when colors are similar, and the seeds anchor a min-cut that slices foreground from background. Add a stroke, re-run the cut, get a cleaner mask.

A close cousin is the random walker family. Picture standing on an unlabelled pixel and taking a random walk across the image, where you are more likely to step between pixels of similar color. The question 'which seed will I most probably reach first — a foreground one or a background one?' assigns that pixel a soft probability. Pixels deep inside the object almost surely reach a foreground seed first; pixels across a strong edge are more likely to reach a background seed. These methods are elegant and need no training data at all, but they only know about low-level cues — color and local contrast. They have no idea what a 'dog' is, so a dog lying on a similarly-colored couch confuses them badly.

The deep-learning era of interactive segmentation fixed exactly that blind spot. The trick is wonderfully simple: take the user's clicks and turn them into extra image channels. For each click you draw a little map the same size as the image — a distance map or a Gaussian 'heat' blob centered on the click that fades with distance — one channel for positive (foreground) clicks and one for negative (background) clicks. You stack these channels onto the RGB image and feed the whole thing into a segmentation network (a U-Net-style encoder–decoder, exactly like guide 2). The network now sees both the picture and where the user pointed, and it has learned from data what objects look like.

The magic is the user-in-the-loop loop. The model predicts a mask; you glance at it; wherever it leaked (grabbed a bit of background) you add a negative click, wherever it missed (left a hole) you add a positive click; the model refolds those new channels in and refines. Each correction click literally tells the network 'you were wrong here' and the next mask snaps closer to what you want. Three or four clicks usually get a clean object — far fewer than painting it by hand.

Segment Anything Model (SAM)

The Segment Anything Model, released by Meta AI in 2023, is best understood as three cooperating parts with one guiding principle: do the expensive work once, make prompting cheap. When you open an image in a SAM-powered tool, it briefly thinks, then lets you click around and get instant masks. That snappiness is not luck — it is the architecture. Let us take the three parts in turn.

Part 1 — a heavy image encoder. This is a Vision Transformer (ViT), the same architecture you met in the transformer track. It runs once per image and is the costly step. Recall how a ViT works: it chops the image into a grid of small fixed-size patches, flattens each patch into a vector (a 'token'), and then lets the tokens talk to each other through self-attention so each one is informed by the rest of the image. The output is a dense grid of feature vectors — a rich image embedding that encodes 'what is where' across the whole picture. Because this embedding is computed once and cached, every subsequent click reuses it for free.

The reason a transformer is the right engine here is attention — every patch token can attend to every other, so a patch on a dog's ear is influenced by the patch on its tail, even though they are far apart. That global reach is how SAM understands an object as a connected whole rather than a bag of local textures, which is exactly the long-range context problem we wrestled with using atrous convolutions and ASPP back in guide 3. Attention gives it for free what those tricks worked hard to approximate.

Part 2 — a lightweight prompt encoder. This small module turns your prompt into a handful of embedding vectors that live in the same space as the image features. A point becomes a positional embedding plus a tiny learned tag saying 'foreground' or 'background'; a box becomes two corner embeddings; a rough input mask is downsampled and added directly to the image embedding. Whatever you give it, the prompt encoder is cheap — it does almost no work compared to the image encoder.

Part 3 — a fast mask decoder. This is where image embedding and prompt embeddings meet. A small transformer decoder lets the prompt tokens and image features attend back and forth a couple of times, then upsamples the result into a full-resolution mask. The whole decode runs in milliseconds, which is the entire point: once the heavy encode is done, each click flows prompt → decoder → mask fast enough to feel live under your cursor. That split — encode once, decode per prompt — is what turns a giant ViT into something a person can play with interactively.

# The encode-once, prompt-many pattern that makes SAM feel interactive
predictor.set_image(image)        # heavy: runs the ViT encoder ONCE, caches embedding

# Now every prompt is cheap — reuses the cached image embedding:
masks, iou_scores, _ = predictor.predict(
    point_coords=[[420, 360]],    # one click (x, y)
    point_labels=[1],             # 1 = foreground, 0 = background
    multimask_output=True,        # return several candidate masks (see next section)
)

# Add a correction click and predict again — still milliseconds, no re-encode:
masks, iou_scores, _ = predictor.predict(
    point_coords=[[420, 360], [500, 410]],
    point_labels=[1, 0],          # second click marks background to remove a leak
    multimask_output=True,
)

How SAM is trained and what 'anything' means

What makes SAM a foundation model rather than just another segmenter is the scale and generality of what it learned. To learn 'anything' you need to have seen almost everything — and no existing dataset was remotely big enough. So the SAM team built one with a clever data engine, a flywheel where the model and the dataset bootstrap each other.

1. Assisted stage: human annotators click on objects and a first, weaker SAM proposes masks; the humans just fix mistakes instead of tracing from scratch — far faster labelling.
2. Retrain: feed those new, higher-quality masks back in and retrain SAM so it gets better at proposing.
3. Semi-automatic stage: the improved SAM auto-masks the confident objects; humans only label the things it missed, focusing effort where it counts.
4. Fully automatic stage: now strong enough, SAM is prompted with a dense grid of points and generates masks for everything by itself, scaling to over a billion masks across 11 million images — the SA-1B dataset.

The second ingredient is promptable pretraining. Instead of training SAM to predict a fixed set of classes, the training task itself is: 'given an image and any prompt, predict the correct mask.' Because the prompts during training are varied and random, the only way to do well is to genuinely understand object structure for all kinds of things — there is no shortcut of memorizing 'car looks like this'. This pretext task forces generality, and the payoff is strong zero-shot transfer: point SAM at a microscope image, a satellite photo, or an underwater scene it never saw during training, and it still segments sensibly, with no fine-tuning. 'Anything' means any object you can indicate with a prompt, in roughly any image.

There is an honest wrinkle SAM faces head-on: ambiguity. Suppose you click once on a person's shirt. Do you mean the shirt? The whole person? The group of people standing together? A single click genuinely cannot tell — all three are valid masks. Rather than gamble on one, SAM predicts several masks at once (typically three, roughly corresponding to part / object / scene-level interpretations) and attaches a confidence score to each, so you — or the surrounding system — can keep the one you meant.

That confidence is the predicted IoU: alongside each mask the decoder outputs a number that is SAM's own estimate of how well that mask would overlap the true object — a guess at the very IoU metric we have used since guide 2. It is not a perfect measure of correctness (it is the model grading its own homework), but it is honest and useful: the system can automatically keep the highest-predicted-IoU mask, or rank the three for a user to pick. Same overlap idea, now turned inward as the model's self-assessment.

Evaluating and composing foundation segmenters

Here is the most important practical fact about SAM: it is class-agnostic. It gives you a beautifully precise mask, but it never tells you what the thing is — no label, no category. For a tool where a human is in the loop ('I clicked, I know it's a dog'), that is fine. But for an automatic pipeline that needs labelled segmentation, SAM alone is only half the answer. The other half comes from composing SAM with a model that supplies meaning.

The composition pattern is simple and powerful: let another model produce the prompts. An ordinary object detector outputs boxes with class names ('dog', box here); feed each box to SAM as a box prompt and you get a labelled, pixel-perfect mask — detection's what married to SAM's where exactly. Even better, an open-vocabulary text-grounding model takes a phrase like 'the dog' and locates a box for it, even for categories no detector was trained on; pipe that into SAM and you have text-driven segmentation. This is exactly the recipe behind systems like Grounded-SAM. You can also run it the other way: take a SAM mask, crop it out, and hand it to an image classifier to name it.

# Grounded-SAM-style composition: text -> boxes -> SAM masks -> labelled segmentation
boxes, labels = grounding_model.detect(image, text_prompt="dog. frisbee.")

predictor.set_image(image)                 # one-time SAM encode
labelled_masks = []
for box, label in zip(boxes, labels):
    mask, iou, _ = predictor.predict(box=box, multimask_output=False)
    labelled_masks.append((label, mask))   # SAM gives 'where', detector gave 'what'

# Now each mask carries a name -> a full semantic/instance segmentation, zero-shot.

How do we know any of this is good? With the exact same overlap metrics that have anchored this whole track. Zero-shot or not, frontier or not, a predicted mask is still scored against ground truth by how well the two overlap. Intersection-over-Union (IoU) measures the overlap ratio; the Dice coefficient from guide 2 measures essentially the same agreement weighted toward the intersection; and when the task is instances-plus-stuff, panoptic quality (PQ) from guide 4 combines recognition and mask accuracy into one number. The metrics did not change at the frontier — only the models did.

The state of the field and where to go next

Step back and look at the whole journey, because it tells one connected story. We began with classical image segmentation: thresholding split pixels by intensity, watershed flooded an image to find boundaries, superpixels grouped similar neighbours, and GrabCut let a box prompt drive a graph cut. These were clever but blind to meaning — they knew color and edges, not objects.

Then deep learning arrived. FCN, encoder–decoder, and U-Net (guide 2) taught a network to predict a dense label for every pixel and to recover sharp boundaries by combining coarse semantics with fine detail. DeepLab's atrous convolutions, ASPP, and PSPNet (guide 3) solved the context problem — seeing an object at many scales at once without losing resolution — and CRFs cleaned the edges. Mask R-CNN and panoptic segmentation (guide 4) added the ability to tell individual object instances apart and then unified 'things' and 'stuff' into one coherent labelling of the entire scene with panoptic segmentation.

And finally, this guide: SAM and promptable foundation models broke the last constraint, the fixed category list. By doing the heavy encoding once and decoding cheaply per prompt, by training on a billion masks through a data-engine flywheel, and by embracing ambiguity with multiple masks and predicted-IoU confidence, SAM turned interactive segmentation into open-world, on-demand segmentation. The arc runs from hand-built color models to a model that segments almost anything you can point at.

Where does the frontier go from here? Four directions are wide open. Video and temporally-consistent segmentation: tracking the same mask across frames so it does not flicker or swap identities — SAM 2 already extends the promptable idea to video. 3D and point-cloud segmentation: carving meaning out of LiDAR scans and depth data for robotics and AR, where 'pixels' become points in space. Efficiency for real-time and on-device: distilling heavy ViT encoders so promptable segmentation runs on a phone or a robot, not a server. And open-vocabulary, text-driven masks: closing the loop so you can simply say what to segment and get a labelled mask, with no detector in the middle.

You have now travelled the full road of image segmentation — from coloring pixels by intensity to prompting a model into segmenting anything in any image. The frontier is not a finish line; it is a launch pad. Pick a direction that excites you — video, 3D, efficiency, or language-driven masks — grab a pretrained foundation model, measure honestly with the metrics you now own, and go segment something nobody has segmented before.