Building a Web-Scale Corpus

The appetite for text

Modern base models are trained on a web-scale corpus: not millions but trillions of tokens, hundreds of times more text than any human could read in a thousand lifetimes. The training data is the single biggest lever on what the finished model knows and how well it writes. A clever architecture trained on garbage will produce garbage; a plain one trained on a clean, diverse corpus can be excellent.

So the corpus team's job is not just to gather text — it is to curate it. The pipeline has four big stages: collect raw data, clean it, remove duplicates, and decide how much weight each source should get. Skip any one and the run suffers. We will walk through each.

Where the raw text comes from

The backbone of most public corpora is Common Crawl — a nonprofit that has spidered the open web for years and releases monthly snapshots of billions of pages, free for anyone to download. It is vast and it is also a swamp: alongside Wikipedia and news you get spam, broken HTML, navigation menus, adult content, and the same cookie banner repeated a million times.

Teams supplement the crawl with higher-trust sources: code repositories, books, academic papers, curated Q&A sites, and reference works. These are smaller but far cleaner per token, and they pull the model toward careful, well-structured language. The mix of sources is a deliberate choice we will return to.

Cleaning: turning a swamp into a corpus

Data cleaning is a stack of filters, each cheap to run and each removing a class of junk. The art is being aggressive enough to raise quality without throwing away good, rare text. A typical pipeline looks like this:

1. Language filter: keep only the languages you intend to train on, scored by a fast language detector.
2. Quality heuristics: drop pages with too few words, too many symbols, broken sentences, or suspicious ratios of punctuation and digits.
3. Boilerplate removal: strip repeated menus, headers, footers, and cookie banners that add no information.
4. Safety and PII filtering: remove the worst toxic content and scrub obvious personal data like emails and phone numbers.
5. Model-based filtering: train a small classifier to recognize "looks like a textbook / good Wikipedia" and keep pages it scores highly.

Each filter is a trade-off. Filter too gently and the model marinates in spam; filter too harshly and you erase dialects, niche topics, and the long tail of knowledge that makes a model interesting. Teams tune these thresholds by training small models and watching which recipe gives the lowest loss on held-out, trusted text.

Deduplication: don't read the same page twice

The web is astonishingly repetitive. The same article is mirrored on dozens of sites; license texts and quotes appear millions of times. Deduplication removes these copies, and it is one of the highest-return steps in the whole pipeline. Why does it matter so much? Duplicated text gets over-weighted: the model sees it again and again, wastes capacity memorizing it, and is more likely to regurgitate it verbatim — which also raises privacy and copyright concerns.

Deduplication happens at two granularities. Exact dedup hashes whole documents and discards identical ones. Fuzzy dedup catches near-duplicates — same article with a different ad in the header — usually with a technique called MinHash that estimates how much two documents overlap without comparing them word for word.

Fuzzy deduplication scores near-duplicates by Jaccard similarity — the overlap between two documents' token sets.

before dedup:  2.4T tokens  (raw, cleaned web)
after exact:   1.9T tokens  (-21%, identical copies gone)
after fuzzy:   1.3T tokens  (-32%, near-duplicates gone)
# nearly half the corpus was redundant

From sources to a single stream

Now you have several clean, deduplicated piles — web, code, books, papers. But the model trains on one stream, so you must decide what fraction of training tokens each pile contributes. That choice is data mixture weighting, and it is surprisingly powerful: upweight code and the model writes better programs; upweight a language and it speaks it more fluently. It is so central that the next guide is partly devoted to it, alongside how many tokens to train on in the first place.