audio language models

Audio language models treat sound the way large language models treat words: as a sequence of tokens to be predicted one after another. By turning speech, music, or ambient audio into discrete codes and training a transformer to continue the sequence, the model can generate or understand audio with the same machinery that powers text generation, producing speech that keeps a speaker's voice and prosody, continuing a melody, or, when coupled to text, following spoken instructions and answering aloud.

The enabling trick is neural audio tokenisation. A residual vector-quantised codec compresses a waveform into a stack of discrete tokens that a transformer can model autoregressively, and many systems separate coarse semantic tokens, which capture content and prosody, from fine acoustic tokens, which capture speaker identity and recording detail, generating them in a hierarchy. Increasingly these audio tokens are folded into a single multimodal language model so that text and speech share one vocabulary and one set of weights, yielding spoken dialogue systems that listen and reply end to end.

Because they model audio at the token level, these models capture long-range structure that earlier signal-processing pipelines missed, but they inherit codec artefacts, can drift over long generations, and raise sharp safety issues around voice cloning. The line between an audio language model and a speech foundation model is blurring as recognition, synthesis, and dialogue converge into one system.