Common Crawl data

Common Crawl is a free, public archive that has been crawling the open web for years, saving billions of pages each month. Because it is enormous and openly available, it forms the raw backbone of many LLM pretraining corpora — when people say a model was trained on the web, this is usually a big part of what they mean.

Straight out of the archive it is almost unusable: raw HTML, boilerplate, machine-generated spam, adult content, and endless duplicates. So teams run it through heavy filtering pipelines — extracting clean text, scoring pages for quality, dropping toxic or low-value material, and removing repeats — to distil a usable fraction. Refined derivatives like C4, RefinedWeb, and FineWeb are all this cleaning applied to Common Crawl.

Relying on it has real consequences: it inherits the web's biases and skews toward English and toward whatever the open web happens to publish. It also raises copyright and consent questions, since it sweeps up text whose authors never agreed to train a model.