JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

From Words to Worlds: What a Multimodal LLM Is

The one-model-many-senses idea, why we bolt vision onto a language model, and the simple trick that makes it work.

A model that does more than read

A plain large language model only ever sees text: you type words, it predicts more words. A multimodal LLM (multimodal large language model) keeps that same talking-and-reasoning core but gains extra senses — most commonly sight. You can hand it a photo, a screenshot, a chart, even a short video, and it answers in words about what it perceives. When the extra sense is specifically images, people call it a vision-language model (VLM).

The mental shift is small but powerful. Instead of describing a picture to the model in words — "there is a red sign with white text" — you simply show it the picture and ask your question directly. The model does the looking. That removes a translation step that used to be lossy, slow, and entirely on you.

Why bolt eyes onto a talker

The world is not made of clean text. Receipts, X-rays, math homework photographed off a whiteboard, UI screenshots, product shelves, memes — most real tasks arrive as pixels. A text-only model is blind to all of it. Giving the model eyes unlocks a long tail of jobs that were previously impossible to automate without a custom computer-vision pipeline per task.

Crucially, you get the language model's reasoning for free on top of perception. The model does not just label what it sees; it can read a chart and explain the trend, look at an error screenshot and suggest a fix, or compare two product photos and say which matches a description. Perception plus reasoning in one box is the whole point.

The trick: everything becomes a sequence of tokens

Here is the elegant idea that makes it all work. A transformer does not actually care that its inputs are words — it processes a sequence of vectors. Text gets turned into those vectors by tokenization. So to add vision, we just turn an image into vectors too — image tokens — and splice them into the same sequence right alongside the word tokens. This blending of different input types into one shared token stream is called multimodal tokenization.

Both text and image patches become the same kind of tokens — vectors the transformer can sit side by side and process.

Diagram: text and an image are split into tokens, mapped to token ids, then to embedding vectors.

Once the picture is a handful of tokens sitting in the sequence, the rest of the machine runs unchanged. The attention layers let word tokens attend to image tokens and vice versa, so by the upper layers the model has a single fused understanding of "a question about this picture." You did not build a new kind of network; you found a way to feed pixels into the network you already had.

Click a word to see what it attends to — the very same mechanism lets word tokens attend to image tokens and back.

Interactive self-attention: selecting a token highlights the other tokens it attends to.

Grounding: connecting words to what is actually there

A good multimodal model does not just hallucinate a plausible answer; it ties its words to the pixels in front of it. That linking is called cross-modal grounding — when the model says "the dog on the left," the phrase is genuinely anchored to that region of the image rather than guessed from context. Grounding is what separates a model that reads an image from one that merely imagines a caption that sounds right.

Grounding in action: a good model highlights the very pixels that drove its answer instead of hallucinating one.

Saliency map highlighting the image regions that most influenced the model's prediction.

Vision is also not the end of the road. The same token-splicing trick extends to sound and motion, and the ambition of an any-modality model is a single system that takes any mix of text, images, audio, and video in — and ideally produces several of those out as well. The rest of this track climbs from images, to the other senses, to that frontier.

What to expect — and what not to

Set honest expectations. Today's vision-language models are excellent at description, reading, and broad scene understanding, and surprisingly good at charts and documents. They are weaker at precise spatial reasoning (exact counts, fine measurements, tiny dense text), and they can confidently misread an image just as a text model can confidently make up a fact. Vision does not cure hallucination; it gives the model more chances to be both right and wrong.

  1. Multimodal LLM = a language model with extra senses, most often sight.
  2. It works by turning images into tokens and letting one transformer mix them with words.
  3. Good models ground their words in the actual pixels; weak ones just imagine a plausible caption.
  4. Expect great description and reading; stay skeptical on exact counts and fine detail.