multimodal large language model

A multimodal large language model (MLLM) is a large language model extended so that it can take in images (and sometimes audio or video) in addition to text, and reason and converse about them. Intuitively, it is a text chatbot that has been given eyes: you can paste a picture into the conversation and ask it to describe the scene, answer questions, read the text in it, explain a chart, debug a screenshot, or write code from a UI mockup. Systems like GPT-4V, Gemini, Claude, Qwen-VL, and the open LLaVA family are MLLMs.

The dominant architecture is a three-part pattern: a vision encoder (usually a CLIP/SigLIP-style ViT) turns the image into feature vectors; a connector translates those features into the LLM's input space; and the LLM does the reasoning and generation, treating the converted image as a sequence of tokens prepended to the text. The connector is the main design choice and comes in a few flavors: a simple linear or MLP projector (LLaVA), a small set of learnable query tokens via a Q-Former (BLIP-2) or Perceiver Resampler (Flamingo) that compress many patches into few tokens, or gated cross-attention layers inserted into the LLM (Flamingo) so text attends to vision without lengthening the sequence.

Training typically has two stages mirrored from the LLaVA recipe: an alignment/pretraining stage on large image-text data that mainly teaches the connector to map visual features into language space, then a visual instruction tuning stage on conversational image-grounded data that produces the helpful assistant behavior. Because the LLM brings broad world knowledge and reasoning, MLLMs generalize remarkably well, but they inherit LLM weaknesses (hallucination, prompt sensitivity) plus vision-specific ones (poor fine spatial precision, weak counting, limited high-resolution detail), which is why resolution handling, OCR, and grounding are active areas of improvement.