JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Beyond Pixels: Audio, Speech, and Video

The same encoder-and-splice trick gives a model ears and a sense of time — plus the new cost that motion brings.

Same trick, new senses

Everything you learned for images carries straight over. Want hearing? Swap the vision encoder for an audio encoder that turns a sound waveform into a sequence of feature vectors, project those into token-shaped vectors via a projector, splice them into the sequence, and run the same multimodal model. The recipe — encode, project, splice — is modality-agnostic. Only the front-end encoder changes.

Whatever the sense, the very same transformer block — norm, attention, residual, feed-forward — processes the shared token sequence.

A transformer block showing layer norm, self-attention, a residual connection, and a feed-forward network.

Hearing: audio LLMs and speech-to-text

An audio LLM takes sound directly as input. That covers a wide range: transcribing speech, but also describing non-speech audio ("a dog barking over rain"), answering questions about a recording ("what genre is this clip?"), or analyzing tone and emotion. Sound is fed in as a spectrogram or learned audio features rather than raw samples, but conceptually it is just another stream of tokens for the model to attend to.

Speech-to-text is the most common audio task and worth calling out on its own. Classic transcription systems just output words; an LLM-based one transcribes and understands, so in one pass it can transcribe a meeting, label who likely said what, and hand you a summary with action items. The grounding still matters — the summary should reflect what was actually said, not a plausible meeting.

Watching: video understanding

Video understanding is the richest sense because it adds time. The standard approach is almost embarrassingly simple: sample frames from the video (say one or two per second), encode each frame just like an image with the vision encoder, and lay all those frame tokens out in temporal order in the sequence — sometimes interleaved with the audio track's tokens. The model then reasons across frames: "summarize this clip," "when does the speaker mention pricing?", "did the person pick up the cup before or after sitting down?"

Frames sampled over time get positional encodings so the model knows their order — that ordering is exactly what 'adds time.'

Positional encoding vectors being added to embedding vectors to mark sequence order.

The new tax: time is expensive

Motion is where costs explode. One image might be hundreds of tokens; a minute of video sampled at two frames per second is 120 images — tens of thousands of tokens — and audio adds its own stream on top. This slams straight into context length limits: you can only fit so much in the window, and attention cost grows with sequence length. Long-video understanding is therefore as much an engineering problem (smart frame selection, token compression, efficient multimodal tokenization) as a modeling one.

N_{\text{tokens}} \;\approx\; \underbrace{T \cdot f}_{\text{frames}}\, t_{\text{frame}} \;+\; T \cdot t_{\text{audio}}

Why motion is the new tax: token count grows with duration times frame rate times tokens-per-frame, and audio piles on its own stream.

  1. Audio LLM: hears sound — transcribes, describes, and reasons about recordings.
  2. Speech-to-text: transcribe and understand in one pass (diarize, summarize, extract).
  3. Video understanding = images over time: sample frames, encode, order, reason.
  4. Watch the budget: time means many tokens — choose your sampling rate deliberately.