Feature Pipelines and Continuous Training

The feature pipeline as a first-class system

A model's predictions are only as good as the features flowing into it, and those features are rarely raw inputs — they are computed: rolling averages, joins against reference tables, embeddings, time-since-last-event counters. Feature pipeline orchestration is the discipline of computing those features reliably and on schedule, with explicit dependencies, retries, and backfills. The orchestrator is a directed acyclic graph (DAG) of tasks: 'compute daily aggregates' depends on 'ingest yesterday's events', which depends on 'validate the raw export'.

Treating feature computation as orchestrated infrastructure — rather than a notebook someone runs by hand — buys you the three properties production demands: idempotency (re-running a task yields the same result), observability (you can see which features are fresh and which are stale), and recoverability (a failed upstream task pauses dependents instead of silently feeding them yesterday's numbers).

Training–serving skew: the silent killer

Here is the bug that humbles every team. At training time you compute features in batch, in Python, over a historical table. At serving time you compute the 'same' features in a low-latency online path, often in a different language, over live data. Any discrepancy between the two paths — a different default for missing values, a timezone bug, a rounding difference — is training–serving skew. The model was fit to one feature distribution and is served another; accuracy quietly bleeds out even though every offline metric looked perfect.

Skew is insidious because it produces no error and no crash — only degraded predictions that monitoring may not catch for weeks. It is also the strongest argument for the architecture in the next section: the surest cure is to make training and serving compute features through the same code path, so 'the same feature' is true by construction rather than by hope.

1. Log served features alongside predictions in production.
2. Periodically recompute those same rows through the training path.
3. Diff the two feature vectors; any nonzero distance is skew, located feature-by-feature.
4. Alert on skew as a first-class production metric, not a one-time check.

Training–serving skew is any nonzero gap between the batch- and online-computed feature vectors for the same input.

The feature store as a consistency contract

The architectural answer to skew is the feature store you met in Volume I, now understood as a consistency contract rather than a cache. A feature store serves features through two faces of one definition: an offline store (large historical tables for training) and an online store (low-latency key-value lookups for serving), both materialized from the same feature definition. Define the feature once; the store guarantees training and serving see the same logic.

The subtlety that separates juniors from seniors is point-in-time correctness. To build a training row for an event at time t, you must join the feature as it was at t — never its current value, or you leak the future into the past (label leakage) and your offline metrics become science fiction. The feature store's offline side does this temporal join for you; reproducing it by hand is where most home-grown pipelines silently cheat.

Point-in-time correctness: a training row uses each feature's value as of the latest update timestamp at or before the event time.

# offline (training): point-in-time join
rows = feature_store.get_historical(
    entities=events[['user_id', 'event_ts']],
    features=['user.spend_7d', 'user.country'])
# online (serving): same definition, low latency
f = feature_store.get_online(
    'user_id', uid,
    features=['user.spend_7d', 'user.country'])

Triggers for continuous training

Even a perfect pipeline serves a model into a moving world. As the input distribution shifts — data and concept drift — yesterday's optimal model becomes today's stale one. A continuous training pipeline automates retraining so the model tracks reality instead of decaying toward it. The key design decision is what triggers a retrain.

1. Scheduled: retrain on a fixed cadence (nightly, weekly). Simple, predictable, but blind to sudden shifts.
2. Drift-triggered: retrain when a monitor reports the live distribution has diverged past a threshold.
3. Performance-triggered: retrain when a business or accuracy metric crosses a guardrail (needs fresh labels).
4. Data-volume-triggered: retrain after N new labeled examples accumulate.

Closing the loop reproducibly

A continuous training loop multiplies the reproducibility stakes from the last guide: every automated retrain must pin a versioned data snapshot, record its lineage, and register a content-addressed artifact, exactly as a human-launched run would. The reward is a self-renewing system whose every iteration is fully auditable — you can always answer 'why did the model change last Thursday?' with 'drift trigger fired at 02:14; retrained on snapshot 9a3f; promoted after passing eval.'

We now have a model that retrains itself as the world moves. But automatic retraining without automatic safe release is just a faster way to ship a bad model. The next guide is about getting new models — hand-built or auto-retrained — into production without betting the business on them.