downstream vs. loss scaling

Scaling laws are cleanest for one number: the pretraining loss, how well the model predicts the next token on held-out text. But nobody ships a next-token predictor for its own sake — they care whether it can answer questions, write code, or pass an exam. Downstream-versus-loss scaling is about the gap between those two: a smoothly falling loss does not always translate into a smoothly rising task score.

Pretraining loss almost always tracks scale as a clean power law. Downstream benchmark performance is messier: it can rise smoothly, jump suddenly (the emergence story), or barely move while loss keeps falling. Part of the disconnect is the metric — converting a continuous probability into a right/wrong score distorts the picture — and part is genuine: some skills need a threshold of underlying competence before they show up at all. Researchers increasingly try to fit scaling laws directly on downstream metrics, or find intermediate quantities that predict them, rather than trusting loss as a proxy.

Lower loss is necessary but not sufficient for a more useful model. Two models with identical pretraining loss can differ sharply on tasks, depending on data mix, fine-tuning, and how the skill is elicited.