multimodal model
/ muhl-tee-MOH-dul MOD-ul /
A multimodal model is one that works with more than one kind of input or output — not just text, but also images, audio, or video, and often a mix. You can show it a photo and ask a question about it, hand it a chart and ask for the trend, or speak to it and get spoken words back. "Modality" just means a type of signal — text, sight, sound — and "multimodal" means the model handles several of them within one system.
The trick that makes this possible is turning every kind of input into the same internal currency: long lists of numbers (embeddings) that live in a shared space, so a picture of a dog and the word "dog" land near each other. Once images, sounds, and words all become number-sequences the same machinery can process, a single model can reason across them — reading the text in a photo, describing a scene, or matching a spoken request to a visual answer. Most leading assistants today are multimodal in this way.
Stay grounded about what this means. Handling images and text together does not give the model human-like perception or understanding; it is still pattern-matching, now across more kinds of patterns. A vision-language model can misread a clock face, miscount objects, confidently describe something that isn't in the picture, or be fooled by an image that a person would never misjudge. More modalities widen what it can attempt — and also widen the surface for confident mistakes.
You photograph the inside of your fridge and ask "What can I cook tonight?" The model recognizes eggs, spinach, and cheese in the image, connects those visual concepts to recipes it learned in text, and replies with a frittata recipe — bridging sight and language in one step.
One model, two senses: it sees the photo and answers in words.
Multimodal does not mean the model perceives like a human. It still pattern-matches — now over images and audio too — and can confidently misread a chart, miscount objects, or describe things not actually present. More senses also mean more ways to be fooled.