masked language modeling
/ MASKT LANG-gwij MOD-ul-ing /
Masked language modeling is a fill-in-the-blank game for training language understanding. Take a sentence, randomly hide a few of its words behind a blank, and ask the model to guess what was hidden: "The cat sat on the ___." To fill that blank well, the model has to use everything around it — both the words before and the words after. This is the key difference from ordinary next-word prediction, which only looks leftward; masking lets the model read in both directions at once.
Why is that two-sided view such a big deal? Because real understanding of a word leans on its full context. To know what "bank" means, it helps enormously to see "river" three words later, not just the words before it. The model BERT, released by Google in 2018, was built on exactly this idea: pretrain a network by making it solve millions of these fill-in-the-blank puzzles over raw text — no human labels needed, the hidden word is its own answer — and it learns deep, context-aware representations of language as a by-product.
This is a form of self-supervised learning, and it powered a wave of huge gains across NLP: a single BERT-style model, once pretrained, could be lightly fine-tuned to do classification, named-entity recognition, question answering, and more. Two honest notes: masked models are built to understand and analyze text rather than to generate long fluent passages (that's the niche of left-to-right generators like the GPT family), and the artificial blank tokens used in training never appear when the model is actually used — a small mismatch researchers have long tinkered with.
Training input: "I went to the [MASK] to deposit my paycheck." The model should predict "bank" — and crucially, the word "deposit" after the blank is what rules out the river meaning. Two-sided context is doing the work.
The word after the blank settles the meaning — only a two-sided reader can use it.
Masked models (BERT-style) read both directions and excel at understanding text, but aren't built to generate long passages — that's the job of left-to-right models like GPT. The two are cousins, not rivals.