parameter-count scaling

Parameters are the model's adjustable knobs — the weights it tunes during training. Parameter-count scaling asks the simplest growth question: if I hold the data fixed and just make the network wider and deeper, with more knobs, how much better does it get? The answer, over many orders of magnitude, is a steady power-law improvement: more parameters, lower loss, in a way you can plot and predict.

Hold tokens fixed and loss falls roughly as N to the minus alpha-N, with alpha-N a small positive number measured from experiments. More parameters give the network more capacity to store patterns, idioms, and facts, so it predicts the next token more sharply. But parameters alone are not free skill: a huge model starved of data overfits or simply wastes capacity, which is exactly the imbalance Chinchilla diagnosed.

Raw parameter count is a seductive headline number, but it is only one of three scaling axes. A 100-billion-parameter model under-trained on thin data can lose to a well-fed 10-billion one.