language modeling
/ LANG-gwij MOD-ul-ing /
At its heart, language modeling is the game of predicting the next word. Given "the weather today is very ___," a language model assigns probabilities to what might come next: "hot" likely, "sunny" likely, "purple" unlikely, "the" almost never. A language model is simply any system that puts probabilities on sequences of words — that can say which sentences sound like real language and which sound garbled. It is the same instinct you use when you finish a friend's sentence before they do.
Why does such a humble task matter so much? Because to predict the next word really well, a system has to quietly pick up a startling amount about grammar, facts, and reasoning. To finish "The capital of France is ___" you need to know geography; to finish "She poured the milk into the ___" you need a feel for physical sense. This is the magic insight behind today's large language models: train a big neural network on nothing but next-word prediction over a huge slice of the internet, and useful knowledge falls out as a side effect — no labels required.
Be honest about what this is and isn't. A language model captures which word sequences are probable, not what is true; it can produce fluent, confident text that is flatly wrong, because plausible-sounding and correct are not the same thing. Older language models counted n-grams; modern ones are neural networks. Either way, the training objective — guess the next token — is the quiet engine under chatbots, autocomplete, translation, and speech recognition alike.
Prompt: "I left my keys on the kitchen ___." A good language model ranks "table" and "counter" high, "sky" near zero. Stack billions of such predictions and you get a system that writes whole paragraphs.
Predict-the-next-word, repeated, is the engine behind modern text generation.
A language model scores plausibility, not truth — fluent text can be confidently false. "It predicts the next word well" and "it knows the facts" are different claims; never assume the second from the first.