language model (basics)

A language model is, in the simplest terms, a system that assigns probabilities to sequences of words. Given the cat sat on the blank, it knows mat is far more likely than helicopter. That sense of which sentences are plausible is all a language model fundamentally has — and it turns out to be enough to power translation, completion, and conversation.

Early language models counted word combinations in text, like how often sat on the was followed by mat. Modern ones replace counting with a neural network that learns the patterns, so they generalize to sentences they have never seen. Either way the job is the same: estimate how likely a stretch of language is, and use that to predict or generate text.