quality filtering of corpora

Raw web text is mostly junk for a language model: spam, navigation menus, machine-generated gibberish, broken encodings. Quality filtering is the pipeline that keeps the tokens worth learning from and discards the rest. Done well it is one of the highest-leverage decisions in pretraining — the difference between a strong model and a mediocre one at the same compute often lives in the filter, not the architecture.

Two layers cooperate. Heuristic filters apply cheap rules: language identification, document length, ratio of symbols to words, fraction of lines ending in punctuation, blocklists, perplexity thresholds from a reference model. Classifier filters are learned: train a binary classifier (often a fast linear model on hashed n-grams, as in the GPT-3 and CCNet pipelines) to score each document by resemblance to a curated high-quality reference set such as Wikipedia or books, then keep documents above a sampled threshold. Recent work (FineWeb, DCLM) uses model-based and even LLM-judged quality scores. The output is a smaller, denser, higher-signal corpus.

The caveat is that 'quality' is defined by the reference set, so filtering imports its biases — over-filtering toward encyclopedic prose can erase dialects, informal registers, and underrepresented topics, narrowing the tail even as it raises average token quality.