deduplication for pretraining

Deduplication is removing repeated text from the corpus before training. The web is astonishingly redundant: the same article is mirrored on hundreds of sites, licences and disclaimers are pasted everywhere, and bots churn out near-identical pages. If you do nothing, the model sees some passages thousands of times and others once, and it over-weights whatever happens to be duplicated.

There are two flavours. Exact deduplication finds documents or long substrings that match byte-for-byte. Fuzzy or near-duplicate deduplication catches passages that are almost the same — using techniques like MinHash to compare documents cheaply at scale. Both shrink the dataset, sometimes drastically, while raising its effective quality because the remaining tokens are more diverse.

The payoff is concrete: deduplicated data trains better models with less compute, reduces verbatim memorisation of training text, and lowers the chance the model regurgitates copyrighted or private passages it saw too many times. It is one of the highest-leverage, lowest-glamour steps in the whole pipeline.