data version control

Git made code reproducible by giving every state of a project an immutable hash you can return to. Data version control extends that promise to the datasets a model learns from, which are too large for Git but every bit as load-bearing for reproducibility. The aim is that any past experiment can be rebuilt exactly because its data snapshot is addressable by version, so the question how was this model trained always has a complete and verifiable answer rather than a best guess.

Because raw datasets can be terabytes, the usual design stores the large files in object storage and keeps only lightweight, content-hashed pointers and metadata under version control alongside the code. Committing a dataset records a hash of its contents; checking out a version restores exactly those bytes. This gives content-addressable, deduplicated snapshots, cheap branching to compare data variants, and a tight coupling between a code commit, a data version, and the resulting model, which together close the reproducibility loop that code versioning alone leaves open.

The discipline pays off in three places: reproducing a result months later, diffing two dataset versions to explain why metrics moved, and satisfying audits that demand the exact training corpus. The friction is operational, large binary snapshots cost storage and require deliberate retention policies, but the alternative, a model trained on a dataset that has since been mutated in place, is a result no one can ever reconstruct or trust.