Dedup first, and mean it
Web pretraining corpora are stunningly repetitive: the same boilerplate, licenses, and copy-pasted articles appear thousands of times. Deduplication is the single cheapest, most reliable win. Exact-match dedup is trivial; the real gain is near-duplicate removal, usually via MinHash plus locality-sensitive hashing (LSH) on document shingles, or suffix-array matching to delete long repeated substrings within documents.
Why it matters beyond saving tokens: duplicated text is what models memorize first, which inflates training loss improvements that don't transfer, worsens privacy leakage, and corrupts evaluation when a test passage sits in the training pile. Dedup improves generalization at fixed compute and is a prerequisite for honest contamination checks later.
Filtering for quality without filtering out diversity
Quality filtering decides which documents are worth training on. Two families combine: cheap heuristics (length, symbol ratios, stop-word presence, language ID, perplexity under a reference model) and a learned classifier trained to recognize text resembling a trusted reference set such as curated reference pages or textbooks. The classifier is powerful but dangerous — it can encode the reference's stylistic and demographic bias and silently delete valid dialects, domains, or tail topics.
Perplexity filtering scores each document under a reference model — lower perplexity means more typical, fluent text.
Pruning: less data, more learning
Beyond removing bad data, data pruning removes redundant data — examples the model already finds easy. Difficulty scores (loss, gradient norm, EL2N, or memorization/forgetting statistics) rank examples; you keep the informative middle and tail. The striking theoretical result is that with a good pruning rule the test error can fall faster than the usual power law, because you are spending compute on the examples near the decision boundary rather than on a thousand easy near-copies.
There is a regime caveat: in the data-scarce regime, keep the easy examples (they stabilize learning); only in the data-abundant regime should you aggressively prune to the hard tail. This connects to coreset selection — choosing a small weighted subset whose training dynamics approximate the full set — and, at the extreme, to dataset distillation, which synthesizes a handful of artificial examples that train a model to near full-data accuracy.
Log-log plot of loss versus scale showing the neural scaling-law power line and a lower curve reachable with aggressive pruning.
Mixing domains: the corpus is a portfolio
A pretraining corpus is a mixture of domains — web, code, reference articles, books, math — and the sampling weights are hyperparameters worth as much as the architecture. Data mixture optimization tunes those weights. Naive proportional sampling overweights whatever domain is largest; you instead optimize weights against a target objective, e.g. minimizing validation loss across a set of downstream domains, not just the biggest one.
The training distribution is a weighted mixture of domain distributions; the weights are hyperparameters as consequential as the architecture.
- Train small proxy models on a few candidate mixtures to fit a cheap loss-vs-weights surface.
- Optimize the mixture weights on that surface against a multi-domain target, often with a group-robust (worst-domain) objective.
- Transfer the chosen weights to the full-scale run, and re-validate, since mixture optima drift with model size.