data scaling

Now hold the model fixed and pour in more text. Data scaling asks: with the same network, how much does reading more lower the loss? Just like parameters, more well-chosen tokens buy a smooth power-law improvement — a model that has seen a trillion tokens predicts more fluently than one that saw a hundred billion, and the gain is predictable enough to budget for.

Hold N fixed and loss falls roughly as D to the minus alpha-D, with D the number of training tokens. More data means more examples of rare words, niche topics, and long-range structure, so the model's estimates sharpen and it memorises less spuriously. Crucially, quality and uniqueness matter as much as raw count: duplicated or low-grade text adds tokens without adding much signal, and can even hurt.

Data and parameters scale together for a reason — feeding a small model unlimited text, or a huge model thin data, both hit diminishing returns fast. And the supply of high-quality human text is finite, which turns this axis into a looming constraint.