web-scale corpus

A web-scale corpus is the mountain of text an LLM eats during pretraining — typically measured in trillions of tokens, the equivalent of millions of books. No single source is that big, so these corpora are stitched together from web pages, books, code repositories, encyclopedias, scientific papers, and forum discussions. The sheer breadth is the point: the model can only know about things that appear in its diet.

Building one is mostly a filtering problem, not a collection problem. Raw web data is overwhelmingly junk — spam, navigation menus, broken HTML, near-duplicate pages — so most of the pipeline is about throwing the bad parts away and keeping the good. The composition (how much code, how much high-quality prose, which languages) strongly shapes what skills the final model has.

Scale also forces hard tradeoffs around quality, deduplication, copyright, and contamination — for example accidentally including the test sets used to evaluate the model. The headline token count tells you almost nothing; how the corpus was cleaned and mixed matters far more.