audio LLM

An audio LLM hears. Instead of, or as well as, reading text and seeing images, it takes sound as input — spoken words, music, a baby crying, a car engine — and reasons about it in language. You can play a voicemail and ask for a summary, hum a tune and ask what it is, or feed a meeting recording and ask who said what. It uses the same trick as vision: turn the signal into tokens the language model can read.

A waveform is first converted to a spectrogram or passed through an audio encoder, often a speech model like the Whisper family, producing a sequence of feature vectors; a projector maps them into the LLM's token space. The model then attends over sound and words together. Some audio LLMs also speak back, pairing the language model with a speech synthesiser so the whole loop — listen, think, reply aloud — runs end to end.

Time is the hard part. Audio is long and dense, so models compress aggressively and can miss brief events, overlapping speakers, or the difference between sarcasm and sincerity that lives in tone.