irreducible loss

Language is not fully predictable. Even a perfect model cannot know whether the next word is cat or dog when both fit — real text carries genuine randomness and information the context simply does not pin down. So as you scale up, the loss does not march to zero; it glides toward a floor. That floor is the irreducible loss: the part of the error no amount of parameters, data, or compute can remove.

In the standard fit, loss is written as a constant E plus power-law terms that shrink with size and data. Those terms can be driven arbitrarily small, but E stays — it is the entropy of the text itself, the inherent unpredictability of language given any context. Scaling buys you the reducible part; the irreducible part is set by the data distribution, not the model. A loss curve flattening is often a sign you are nearing E, not that scaling has failed.

E is not universal — it depends on the dataset, the tokenizer, and how much context the model sees. Cleaner, more structured data has a lower floor, which is one more reason data quality, not just quantity, shapes the limit.