zero-shot image classification

Zero-shot image classification means classifying an image into categories for which the model has seen zero labeled training examples. A normal classifier is trained on a fixed set of classes (say the 1000 ImageNet labels) and can only ever output one of those. A zero-shot classifier, by contrast, can be told the candidate categories at test time, in plain words, and assign the image to one — even a category nobody trained on. This is possible because a vision-language model like CLIP has learned a shared space where images and text descriptions are comparable.

The mechanism is direct. You write each candidate label as a text prompt — typically 'a photo of a {label}', e.g. 'a photo of a golden retriever' — and run all the prompts through the text encoder to get one vector per class. You run the image through the image encoder to get its vector. The predicted class is simply the one whose text vector has the highest cosine similarity to the image vector. You can swap in any label set you like at inference with no retraining; you are essentially turning classification into a retrieval problem against text labels.

Two practical levers strongly affect accuracy. First, prompt engineering: the wording matters, and a bare label ('cat') usually does worse than a templated prompt ('a photo of a cat'), because the latter matches the distribution of CLIP's training captions. Second, prompt ensembling: average the embeddings of many templates ('a photo of a {label}', 'a blurry photo of a {label}', 'a sculpture of a {label}', etc.) to get a more robust class vector. CLIP famously reached strong ImageNet zero-shot accuracy this way, and the approach generalizes to fine-grained, satellite, and medical domains with carefully written prompts.