training-data deduplication

Web-scraped corpora are full of the same text appearing many times — boilerplate, mirrored pages, quoted passages, licensing footers. Deduplication removes these exact and near-duplicate copies before training. The motivation is threefold: duplicates waste compute on content the model has already seen, they inflate the effective weight of whatever happens to be repeated, and they sharply increase verbatim memorization, which hurts both privacy and generalization.

Exact duplicates are caught by hashing whole documents or fixed-length substrings. Near-duplicates need approximate methods: MinHash with locality-sensitive hashing estimates Jaccard similarity over shingled n-grams so that documents sharing many n-grams collide into the same bucket, and suffix-array methods remove long repeated substrings within and across documents. The empirical finding (Lee et al. and follow-ups) is that deduplicated training data lets models reach the same or better validation loss with fewer steps and emit far less memorized training text at generation time.

The subtle part is choosing the similarity threshold and granularity: too loose and you delete legitimately similar but distinct examples (and shrink the tail); too strict and duplicates leak through. Dedup also interacts with contamination detection, since a benchmark item duplicated into the corpus is both a duplicate and a leak.