llava

LLaVA (Large Language and Vision Assistant), from 2023, is a simple and influential recipe for giving an existing chat LLM the ability to see. The design is deliberately minimal: take a pretrained CLIP vision encoder (ViT-L/14) to turn an image into visual features, take a pretrained instruction-tuned LLM (originally Vicuna), and connect them with a small projection module that maps visual features into the LLM's token embedding space. The projected image features become 'visual tokens' that sit in the prompt alongside text tokens, and the LLM treats them like any other input it can attend to.

In the original LLaVA the connector is a single linear layer; LLaVA-1.5 upgraded it to a small two-layer MLP and added higher-resolution inputs and academic VQA data, giving large gains for tiny extra cost. The whole point is parameter efficiency and simplicity: most capability already lives in the frozen-or-lightly-tuned CLIP encoder and the LLM, and only the lightweight projector (and optionally the LLM) needs training to align the two.

Training proceeds in two stages. First, feature alignment pretraining: on image-caption pairs, train only the projector so visual tokens land in a region of embedding space the LLM can interpret. Second, visual instruction tuning: fine-tune on conversational, image-grounded instruction data — famously generated by prompting GPT-4 (text-only) with image annotations to produce diverse question-answer dialogues — which teaches the model to follow visual instructions and chat about images. LLaVA showed that this lightweight approach yields a capable open multimodal assistant, and it became the de facto open baseline for multimodal LLMs.