Deduplication

Deduplication is finding and removing repeated records that refer to the same real-world thing, so each entity is counted once. Duplicates creep in when data is merged from several sources or a form is submitted twice.

The hard part is defining what “the same” means: exact-copy rows are easy, but “Bob Smith” and “Robert Smith Jr.” at the same address may be one person recorded two ways (fuzzy matching). Removing real duplicates prevents double-counting that inflates totals; removing the wrong rows silently deletes real data, so always dedupe on a well-chosen key.