Reproducibility: Versioning Models, Data, and Metadata

Why lineage is the foundation

In Volume I you learned to train a model and measure it. In production the first question is rarely *how accurate is it?* — it is *which exact model is serving this request, and can I rebuild it from scratch?* A model in production is not a file; it is the deterministic product of three coordinates: the code that defined the architecture and training loop, the data it was trained on, and the configuration (hyperparameters, seeds, library versions). Lose any one coordinate and you have an artifact you can run but cannot explain, audit, or fix.

Model versioning is the discipline of binding those three coordinates together so every released model carries its full provenance. It is the unglamorous backbone that makes every later chapter — safe release, rollback, monitoring, governance — even possible. You cannot roll back to a 'known-good' model if you cannot name it; you cannot debug a regression if you cannot diff two versions; you cannot pass an audit if you cannot show the data lineage.

Versioning the model artifact

Naive versioning bumps a number every time someone clicks 'export'. Production versioning instead computes an immutable, content-addressed identity for each model and stores it in a model registry — a catalog that records the artifact, its hash, its training run, its metrics, and its lifecycle stage (staging, production, archived). The registry is the single source of truth for 'what may be deployed'; the serving layer only ever pulls a model by registry reference, never by ad-hoc file path.

The decisive move is to make the version include the inputs, not just label the output. A robust version key is a hash over (code commit + data snapshot id + config). Two trainings with the same key must yield the same model; if they don't, you have hidden nondeterminism (unseeded shuffling, nondeterministic GPU kernels, race conditions in data loading) that you must hunt down — because nondeterminism you cannot see is nondeterminism you cannot debug.

A content-addressed version key: hashing the code commit, data snapshot, and canonicalized config collapses identical inputs to the same id.

version_key = sha256(
  code_commit_sha +
  dataset.snapshot_id +
  canonical_json(config)        # sorted keys, fixed float fmt
)
registry.register(
  artifact=model_bytes,
  key=version_key,
  metrics={'auc': 0.94, 'val_loss': 0.21},
  lineage={'data': dataset.snapshot_id,
           'code': code_commit_sha},
  stage='staging')

Versioning the data

Code versioning is solved; data versioning is where teams quietly fail. Datasets are large, mutable, and constantly appended to. Data version control gives you the same guarantees git gives code: an immutable snapshot id that names exactly the rows and files used, without copying terabytes around. The standard trick is to version pointers and hashes in git-like metadata while the bulk bytes live in object storage, deduplicated by content hash.

1. Snapshot the dataset: compute a content hash over the file manifest (path → blob hash) and freeze it as a snapshot id.
2. Store only changed blobs; unchanged files are shared across snapshots, so a small append costs little.
3. Reference the snapshot id (never a mutable 'latest' path) in every training run's lineage record.
4. Re-resolving a snapshot months later must return byte-identical data — that is the contract.

Versioned data also lets you diff datasets: when last week's model beat this week's, the most common culprit is a silent data change — a relabeled batch, a dropped feature column, an upstream schema migration. With snapshots you can answer 'what changed in the data?' as crisply as 'what changed in the code?'.

The metadata store: the system of record

Versioning produces facts — runs, params, artifacts, metrics, lineage edges. ML metadata tracking is the queryable database that stores these facts and the relationships between them, turning a pile of files into a graph you can interrogate. It generalizes the experiment tracking you met in Volume I from 'a dashboard of my training curves' into 'the audit-grade ledger of everything the ML system ever produced'.

The graph is what makes hard questions cheap: *which models were trained on the dataset we must now delete for privacy reasons?* is a single traversal from a data node to all downstream model nodes. Without the graph it is a frantic spreadsheet archaeology dig during a compliance deadline.

The privacy question as graph reachability: every model node reachable from a data node is its downstream lineage.

Tying it together with CI/CD

Reproducibility is only real if it is automated. CI/CD for ML turns the versioning discipline into pipeline gates: a commit triggers a training pipeline that pins the data snapshot, computes the version key, registers the artifact, runs the evaluation suite, and only promotes the model to the 'staging' stage if it clears the bar. Humans approve promotions; the pipeline enforces that nothing un-versioned can ever reach production.

The mindset shift from Volume I is this: a model is no longer the output of your work, it is one immutable, named node in a lineage graph that the rest of the operations stack — release, monitoring, governance — will reference by name for the rest of its life. Get this layer right and everything downstream becomes tractable. The next guide builds upward from a single training run to a continuously retraining pipeline.