data pruning and scaling

Naively, more data helps but with diminishing returns: error falls as a slow power law of dataset size, so each doubling buys less. Data pruning asks a sharper question — what if you throw data away on purpose, keeping only the most useful examples? Done well, you can bend that power law, reaching the same accuracy with far fewer examples, or better accuracy at the same budget. The slogan from the Sorscher et al. result is that good pruning can beat power-law scaling.

The mechanism rests on an example-difficulty score, typically how hard or uncertain each example is for the model (early-training loss, margin, forgetting events, or self-supervised prototype distance). The optimal pruning policy is data-abundance-dependent: when data is scarce keep the easy, prototypical examples to nail the basics; when data is abundant keep the hard, informative ones near the decision boundary and discard easy redundancy. This adaptive rule is what turns pruning from a constant-factor saving into a change in the scaling exponent itself.

In practice for large language and vision corpora, pruning interacts with deduplication and quality filtering, and the difficulty scores are noisy and seed-dependent. Prune too aggressively and you delete the tail you needed; the gains are real but require validating the score on your own distribution.