Data Orchestration

Orchestration is the scheduling and coordination of all the steps in your data pipelines: deciding what runs, in what order, when, and what happens when a step fails. An orchestrator is the conductor — it does not usually crunch the data itself, it tells each task when to start and makes sure step B waits for step A to finish successfully.

This matters because real pipelines have dozens of interdependent tasks, and getting the order and timing wrong produces stale or half-broken data. Orchestrators model the work as a DAG (directed acyclic graph) — a flowchart of tasks with no loops — and handle retries, alerts, and backfills (re-running history). Apache Airflow is the classic example; Prefect and Dagster are popular modern alternatives.