speech-to-text with LLMs

Speech-to-text — transcription — is the most established audio task: turn spoken words into written ones. It powers captions, dictation, voice assistants, and meeting notes. Plain transcription has existed for years, but routing it through an LLM adds understanding: the model can punctuate sensibly, label speakers, translate as it transcribes, and answer questions about what was said instead of just dumping a wall of text.

The audio encoder turns speech into feature vectors and the language model decodes them into text, predicting words one at a time exactly as it does in writing. Because the LLM carries world knowledge and grammar, it disambiguates homophones from context — their versus there — recovers proper nouns, and keeps formatting clean, advantages a pure acoustic model lacks. The same architecture handles many languages and code-switching within a single utterance.

Accuracy collapses with noise, accents, jargon, and crosstalk, and the model can hallucinate fluent text during silence or unclear audio — inventing plausible words that were never spoken, which is more dangerous than an honest blank.