dataset distillation

Dataset distillation compresses an entire training set into a handful of synthetic examples — often just one or a few per class — such that a model trained from scratch on those synthetic points reaches nearly the accuracy of one trained on the full data. Unlike a coreset, which subsamples real examples, the distilled images or token sequences are optimized artifacts; they may look like ghostly blends of many real images and need not resemble any single one.

It is posed as a bi-level optimization: the inner loop trains model parameters on the small synthetic set, the outer loop adjusts the synthetic data to minimize loss on the real data after that inner training. Because differentiating through full inner training is expensive, practical methods use surrogates — gradient matching (make synthetic-data gradients mimic real-data gradients each step), trajectory matching (align the parameter path), distribution matching in feature space, or kernel-ridge-regression closed forms (KIP). The synthetic set, plus sometimes learned soft labels and learning rates, is the distilled dataset.

Applications include fast neural architecture search, continual learning with tiny replay buffers, and privacy-flavored data sharing. The honest limits: distilled sets are tuned to a model class and degrade across architectures, scaling to high resolution and to many classes remains hard, and 'privacy' is empirical, not a formal guarantee.