Beyond Images: Audio, Speech, Video, and Any-to-Any

One recipe, many modalities

Everything in the last three guides — align a modality to language, bridge a frozen encoder into an LLM, instruction-tune — is not specific to pixels. Swap the vision tower for an audio or video encoder and the architecture barely changes. The deep reason is that the bridge speaks in tokens: once any signal is turned into a sequence of embeddings, a transformer does not care whether they came from patches, spectrogram frames, or video clips. This is why progress in vision-language models transferred so quickly to sound and motion.

Speech foundation models

Speech foundation models are the audio analogue of CLIP and BERT: large encoders trained with self-supervised objectives on huge unlabeled (or weakly labeled) audio. Some, like wav2vec/HuBERT, mask and predict latent units of the waveform; others, like Whisper, train weakly-supervised on hundreds of thousands of hours of transcribed audio scraped from the web, which yields a single model that transcribes and translates across dozens of languages with striking robustness to noise and accent.

\mathcal{L}=-\log\frac{\exp\!\big(\mathrm{sim}(c_t,q_t)/\tau\big)}{\sum_{\tilde q\in Q_t}\exp\!\big(\mathrm{sim}(c_t,\tilde q)/\tau\big)}

Speech foundation models learn by contrast: pull the context vector toward the true quantized target and push it away from distractors.

Audio language models

Audio language models go further than recognition: they treat audio as a language to generate, not just perceive. The key enabler is a neural audio codec — a vector-quantized VAE that compresses a waveform into a short sequence of discrete acoustic tokens. Now speech synthesis becomes autoregressive next-token prediction over those codes, exactly like text. A model can be prompted with three seconds of a voice and continue speaking in it, or take spoken input and reply with spoken output, end to end.

\mathbf{z}_q=\mathbf{e}_{k},\qquad k=\arg\min_{j}\;\lVert\mathbf{z}_e(x)-\mathbf{e}_j\rVert_2

A neural audio codec quantizes each encoder output to its nearest codebook entry, turning a waveform into discrete tokens a model can generate.

Text to video

Text-to-video generation is image generation plus the hardest extra constraint: time. A believable clip must keep objects, identities, and physics coherent across frames while obeying the prompt. The dominant approach extends a diffusion model into space-time — denoising a compressed latent video with attention that mixes spatial and temporal axes. Modern systems run the diffusion transformer over a latent built by a video autoencoder, so the model never denoises raw pixels frame by frame.

Any-to-any: one model that perceives and generates

The unifying ambition is any-to-any multimodal models: a single network that takes any mix of text, image, and audio in, and produces any mix out. The cleanest formulation tokenizes every modality into a shared discrete or continuous vocabulary and runs one transformer over the lot — understanding and generation become the same next-token problem. Architectures vary: some are purely autoregressive over discrete tokens, others couple an autoregressive backbone to a diffusion head that renders continuous pixels or audio for higher fidelity.

Unifying modalities sharpens an old problem. When a single space must host pixels, phonemes, and words, the modality gap from guide 1 returns with a vengeance, and balancing the modalities so one does not dominate training is delicate. The next guide takes these tensions head-on: how do we measure whether a multimodal model is actually any good?