data mixture weighting

A pretraining corpus is a blend of many sources — web text, books, code, math, Wikipedia, scientific papers — and mixture weighting is deciding how much of each the model actually sees. It is the recipe rather than the ingredients: you might hold a trillion tokens of code but choose to sample it as ten percent of training, while upweighting a small, high-quality book collection so it punches above its raw size.

These weights are tuning knobs with real effects. More code tends to improve reasoning and structured output; more high-quality prose improves fluency; more multilingual text broadens language coverage. Because some sources are small but valuable, they are often sampled multiple times per pass, while vast low-quality sources are downsampled. Teams pick weights with small proxy experiments and scaling-law extrapolation rather than guesswork.

The hard part is that the best mixture depends on what you want the model to be good at, and the choices trade off against each other. Curriculum ordering — showing easier or cleaner data early and harder data later — is a related lever layered on top of the static mixture.