causal language modeling

Causal language modeling means the model reads strictly left to right and may only look at words that came before, never after. Picture covering a page with your hand and sliding it rightward one word at a time, predicting each new word from what is already revealed. The word causal captures this one-way flow: the past can shape the next token, but the future is invisible.

Mechanically, a mask inside the attention layers blocks every position from attending to later positions, so the prediction for token t depends only on tokens 1 through t-1. The whole sequence is still processed in parallel during training, but the mask guarantees no peeking ahead. This is exactly the setup that lets the trained model generate: feed it a prompt and it extends it one token at a time.

It contrasts with masked language modeling (as in BERT), which hides scattered words and lets the model use both sides for context. Causal models trade that two-sided view for the ability to write fluent continuations — which is why nearly every generative LLM is causal.