JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Speaking in Pictures: Generation and the Any-Modality Frontier

Letting the model output more than text — image generation, any-to-any models, and an honest look at where multimodal is heading.

Closing the loop: output, not just input

Everything so far has been multimodal in, text out. The frontier closes the loop: models that also produce non-text modalities. The headline case is image generation from LLMs — you describe a picture and the model draws it, or you give it a photo and a request ("make it sunset, keep the dog") and it edits. The same language understanding that lets a model read an image is exactly what lets it control the creation of one.

Generation closes the loop: the model emits its output one token at a time in an autoregressive loop.

Diagram of an autoregressive generation loop feeding each predicted token back as the next input.

How an LLM ends up drawing

There are two common wirings. In the first, the LLM acts as a conductor: it interprets your messy request, writes a precise prompt, and hands that to a separate image generator (often a diffusion model). The LLM never paints a pixel — it directs an artist. In the second, more unified design, the model emits special image tokens of its own that a decoder turns back into pixels, so generating a picture is literally the same next-token loop as generating a sentence — a direct payoff of multimodal tokenization.

In the conductor design, the LLM's prompt drives a separate diffusion model that denoises pure noise into an image.

Diffusion process turning random noise into a clear image step by step.

Any-to-any: one model, every direction

Push that idea to its limit and you get the any-modality model: a single system that accepts any mix of text, image, audio, and video, and can emit several of those too. Speak a question about a photo and get a spoken answer; hand it a chart and ask for a redesigned version; feed a silent clip and request a narration. Because each modality is just tokens in a shared stream, the model can in principle translate between any pair of senses through that common interlingua.

p(x_{1:T}) = \prod_{t=1}^{T} p\!\left(x_t \mid x_{<t}\right)

An any-modality model factorizes one unified token stream autoregressively, whatever the modality of each token.

This is also where cross-modal grounding becomes central rather than nice-to-have: an any-modality model is only trustworthy if its generated audio actually says what its read image showed, and its drawn picture actually matches the words it was given. Generation makes grounding failures visible and sometimes embarrassing — a model can now draw its hallucination, not just describe it.

Where this is heading — and honest limits

The trajectory is clear: from text-only, to read-an-image VLMs, to hearing and watching, to generating across modalities, to one any-modality system that fluidly mixes them all. Each step reuses the same encode-project-splice plumbing you learned in guide two; the frontier is mostly about doing it for more senses, at higher resolution, over longer spans, more efficiently.

Stay grounded about the gaps. Fine spatial precision, reliable counting, tiny dense text, long-video reasoning under a tight context budget, and faithful generation that never drifts from the prompt — these are all still hard. Multimodal LLMs are genuinely transformative for perception-plus-reasoning tasks, and they will still confidently misread a blurry label. Treat them as a brilliant, fast, occasionally-wrong assistant with eyes and ears — and verify anything that matters.

  1. Generation closes the loop: models output images (and more), not just read them.
  2. Two wirings: LLM as conductor of a separate generator, or unified image-token output.
  3. Any-to-any is the destination — every modality as tokens in one shared stream.
  4. Grounding gets harder and more visible once the model generates; verify what matters.