curriculum data ordering

We teach children arithmetic before calculus, and curriculum data ordering applies the same idea to training: present examples in a meaningful sequence — typically easy first, then progressively harder — instead of one uniformly random shuffle. The hope is faster convergence, a better final optimum, and more stable training, because early easy examples carve out useful representations before the model has to wrestle with the hard, ambiguous, or noisy cases.

Making it concrete requires a difficulty measure and a pacing function. Difficulty can be model-independent (sentence length, rarity, human-rated complexity) or model-derived (current loss, gradient norm, learning speed, confidence). The pacing function controls how fast the harder portion of the pool is unlocked over training. Variants invert the intuition: anti-curriculum (hard-first) and self-paced learning, where the model itself decides which examples are ready, treating example weights as latent variables optimized jointly with parameters. For large language models, related strategies order or upweight data by quality, domain, or sequence length across training phases.

Evidence is mixed and task-dependent: curricula help most when the task is hard, the data is noisy, or the model is small, and can be neutral or harmful otherwise — well-tuned random shuffling is a strong baseline that curricula must actually beat.