data cleaning for LLMs

Data cleaning is the unglamorous work of deciding what text deserves to be in the training set. Raw web data is mostly noise — menus and footers, gibberish, ad copy, broken encodings, hateful or explicit material — and a model trained on slop learns to produce slop. Cleaning is the filter between the open internet and the model's mind, and it quietly determines a lot of the final quality.

A typical pipeline strips HTML to plain text, detects and keeps target languages, removes boilerplate, and then scores each document. Scoring can be rule-based (too short, too many symbols, too repetitive) or model-based (a small classifier trained to recognise clean, useful prose). Toxicity and personally identifying information are filtered too. Documents that fail the bar are simply dropped — often the majority of the raw data.

The danger is over-cleaning. Filters that chase a narrow notion of quality can erase dialects, minority languages, and informal voices, baking the curators' taste and blind spots into the model. Cleaning is never neutral — it is an editorial decision at planetary scale.