multimodal LLM

An ordinary LLM only reads and writes text. A multimodal LLM also takes in things that aren't text — a photo, a screenshot, a chart, a clip of speech, a few seconds of video — and reasons about them in the same conversation. You can paste a picture of your fridge and ask what to cook, or hand it an X-ray, a maths worksheet, or a song. The model treats the image or sound as just more context sitting alongside your words.

Under the hood it stays a next-token text model. A separate encoder turns the non-text input into a sequence of vectors, a small projector reshapes those vectors so they sit in the same space as word embeddings, and they are spliced into the prompt as if they were extra tokens. The transformer then attends across words and image patches together. Only a thin bridge is new; the language brain is reused.

Be careful with the word: multimodal usually means understanding extra modalities, not generating them. Most of these models still only output text, and making pictures or audio needs a separate generator. They also inherit every text-LLM weakness — they can confidently misread a chart or hallucinate a detail that isn't in the image.