What Vision-Language Models Can Actually Do

From describing to doing

Once a model can see, a whole menu of tasks opens up. They sit on a ladder of difficulty: the simplest is just describing an image; the richest is following a multi-step instruction about it. The same model usually does all of them — the difference is what you ask, not which model you load. This guide walks the ladder rung by rung so you know what to expect and how to phrase the request.

Captioning and visual Q&A

Image captioning is the entry task: "describe this image." It is great for alt-text, content moderation triage, and bulk tagging. You can steer the style — "one short sentence," "list every object," "focus on the mood" — because the caption is just generated text and obeys your instructions like any other prompt.

Visual question answering (VQA) goes further: you ask a specific question and the model answers from the image — "How many people are wearing hats?", "Is the light red or green?", "What brand is this?" VQA is where grounding earns its keep: a good answer points at the right region rather than guessing from priors. It is also where hallucination shows up — if the answer is not visible, a weaker model may invent one rather than say "I can't tell."

Following visual instructions

Visual instruction following is the capability that turned narrow caption-and-VQA models into general visual assistants. After instruction tuning on examples that pair an image with a free-form request and a helpful answer, the model will do open-ended things: "Rewrite the sign in this photo in formal English," "From this whiteboard, extract the action items as a checklist," "Explain this diagram to a five-year-old." It is the visual analogue of going from a text completer to a chat assistant.

messages = [
  { role: "user", content: [
      { type: "image", source: photo_of_receipt },
      { type: "text",  text: "Extract every line item and total as JSON. "
                            + "If a price is unreadable, use null." }
  ]}
]
# The model reads the receipt AND obeys the formatting instruction.

Reading: OCR, charts, and documents

A huge fraction of real value is just reading images of text. OCR with LLMs (optical character recognition) folds classic text extraction into the model: it transcribes printed pages, handwriting, signs, and screenshots — and because a language model is doing it, the output can be cleaned, translated, or summarized in the same breath, not just dumped as raw characters.

Chart and document understanding is the high end of reading: not just transcribing text, but interpreting structure. Ask the model to read a bar chart and state the trend, pull a number from a specific cell of a table, answer a question that requires combining the form fields with the fine print, or summarize a multi-column PDF page. This is where vision-language models replace brittle, hand-built document pipelines — though for tiny dense tables you should still verify the numbers.

Choosing the right rung

1. Need a general description or alt-text? → captioning, and steer the style.
2. Need one specific fact from the image? → VQA, and give the model permission to say "not visible."
3. Need an open-ended task done to the image? → visual instruction following, with a clear request.
4. Need to read text, charts, or forms? → OCR / document understanding, and feed high resolution.