From model-centric to data-centric
For a decade the default research loop fixed the dataset and searched over architectures, optimizers, and losses. Data-centric AI inverts that loop: hold a competent model fixed and systematically iterate on the data instead. The wager is empirical — on most mature benchmarks the marginal example, once you have a reasonable model, teaches more than the marginal layer. You are no longer asking what function class fits this data, but what data makes this function class behave.
This is not a rejection of scale. Scaling laws still hold, but they describe loss as a function of tokens and parameters assuming a fixed data distribution. Change the distribution — dedup it, filter it, remix it — and you shift the entire curve, often buying the effect of an order of magnitude more compute. The bitter lesson says general methods that leverage computation win; data-centric work is how you decide what that computation gets to see.
Log-log plot of loss decreasing as model size and data scale grow.
Making 'quality' measurable
The word quality is useless until it is operationalized. Data-quality metrics decompose a vague gut feeling into measurable axes: label noise rate, coverage of the input space, redundancy, class balance, and agreement between annotators or sources. Each axis is something you can estimate, track over time, and tie to a downstream effect. A dataset with 5% label noise and heavy near-duplication is not 'a bit worse' than a clean one — it can cap achievable accuracy regardless of model size.
- Pick a target downstream metric (accuracy, calibration, a benchmark suite) so every data decision has a measurable payoff.
- Estimate the noise and redundancy axes on a held-out clean slice, not on the training set itself.
- Run a small data ablation: train on the raw set vs. a quality-controlled subset of equal size, and compare.
- Attribute the gap to a specific axis, then invest your effort where the elasticity is largest.
A data intervention's true effect is the metric gap measured at a fixed example budget B — so quality is never confounded with quantity.
The leverage argument
Why does data move the needle so much? Because most large datasets are not on the efficient frontier: they contain duplicates, mislabeled tails, and large redundant clusters that contribute almost nothing to generalization yet cost real compute. Data pruning results show you can sometimes beat the power-law scaling curve — reach the same loss with fewer, better examples — which is impossible if you only twist the model.
Data is also where distribution shift is born and where it can be cured. If deployment differs from training, no clever loss recovers the missing signal; you fix it by changing what is in the set. That is the recurring shape of this whole track: diagnose with metrics, intervene on data, measure the downstream delta.
Distribution shift made precise: training and deployment draw from different joint distributions — a gap no loss function can close, only better data can.
What this track will build
We climb from diagnosis to a self-improving loop. Next we learn to find and fix label errors with confident learning and programmatic labeling. Then we curate web-scale corpora with deduplication, quality filtering, and mixture optimization. We assign credit with Shapley valuation and influence functions. Finally we close the loop with active acquisition and curriculum ordering.