data mixture optimization

A pretraining corpus is a blend of domains — web text, code, books, encyclopedic articles, math. The mixture weights say what fraction of training tokens to draw from each. These weights matter enormously: too little code and the model can't program, too much narrow web text and it loses general knowledge. Data mixture optimization is the problem of choosing those sampling proportions to get the best model for your goals, rather than just using the natural proportions the internet happens to provide.

Several principled approaches exist. Group distributionally robust optimization (DRO) and DoReMi train a small proxy model to find weights that minimize worst-case excess loss across domains, on the theory that protecting the hardest domain yields a robust general model. Others fit a scaling-law surrogate that predicts validation loss as a function of mixture weights and token budget, then optimize the predicted loss; online methods adjust weights during training from per-domain learning signals. The output is a weight vector on the simplex over domains, sometimes annealed over training so that high-quality domains are upweighted late.

The difficulty is that the best mixture depends on model size, total token budget, and which downstream tasks you score — there is no universal recipe, and proxy-model weights do not always transfer cleanly to the full-scale run.