Orchestration & Pipelines
A DAG half-succeeded at 3am: some tasks ran twice, a backfill double-counted revenue, and now you are reconciling numbers instead of sleeping.
This stratum names the failure modes before the tools: the Airflow scheduler loop and why a task ran (or didn't, or twice), idempotency and state, backfills that don't double-count, freshness SLAs that page for the right reasons, and the gap between a green pipeline and correct data.
What you'll learn
- Explain the Airflow scheduler loop and why a task ran, skipped, or duplicated
- Design idempotent tasks and backfills that survive partial failure
- Distinguish pipeline health from data quality — and alert on the right one
- Run Airflow in production: deploys, secrets, logging, multi-tenancy, and cost
Tracks & courses
Full navigation is in the sidebar. Here's what each track gives you and the courses inside it.
Pipeline Failure Modes
Before any tool, learn the shapes of failure. The diagnostic vocabulary every data engineer earns the hard way at 3am.
Anatomy of a Broken Pipeline
Eight named failure modes every on-call data engineer will see. Learn to recognize silent success, schedule drift, partial loads, retry-induced corruption, and the deceptively green DAG.
8 ch · 1h 53m
1 freePipeline Quality vs Data Quality
Pipeline health and data health are independent axes. Learn to tell the difference between 'the job ran' and 'the data is right', and which one belongs on which dashboard.
6 ch · 1h 17m
1 freeOrchestration Vocabulary
Tool-agnostic shared language so Airflow, Dagster, Prefect, and Argo conversations all parse the same. The vocabulary is the durable skill; the tool is rented.
DAGs, Tasks, and the Scheduler Loop
The tool-agnostic vocabulary every orchestrator's docs assume you already know. DAG, task, operator, executor, sensor, scheduler heartbeat. Once you have these words, every orchestrator's docs read the same.
7 ch · 1h 28m
1 freeIdempotency and State
What 'safe to retry' actually means, where to keep state so retries don't lie, and why exactly-once is a marketing term. The six concepts senior engineers reach for without thinking.
6 ch · 1h 13m
1 freeAirflow Internals
Airflow is the default in 2026, and the default is also the most-misunderstood. What the scheduler, executor, and metadata DB are actually doing, and which knob to turn first.
Scheduler Deep Dive
What Airflow's scheduler, executors, metadata DB, and concurrency knobs are actually doing. Every Airflow shop hits the same five performance walls. This course names them before you hit the third one.
10 ch · 2h 4m
1 freeAirflow in Production
Deploying DAGs without restarting anything, secrets that survive failures, upgrades without losing a weekend, multi-tenant survival, and the cost story nobody tells you. The seven production lessons that decide whether Airflow is a tool or a tax.
7 ch · 1h 23m
1 freePipeline Trust, SLA & Lineage
The move from 'the job ran' to 'the pipeline met its contract, on time, and someone owns it'. The orchestration-adjacent slice of observability: freshness SLAs, lineage-aware alerting, and the on-call hygiene that decides whether your team gets paged at 3am for the right reasons.
Freshness and SLA
Freshness is a contract, not a metric. SLA, SLO, and SLI untangled, the cheap way to measure freshness, lineage-aware alerting that routes the root cause, page severity that protects your engineers, and the owner metadata and runbooks that make on-call survivable.
7 ch · 1h 20m
1 freeLineage for Orchestration
Why orchestration needs lineage at all, OpenLineage as an event stream, column-level vs table-level lineage and when each matters, backfill blast radius, lineage as an SLA input, and the three kinds of drift no single tool catches.
6 ch · 1h 6m
1 freePipeline Design Patterns
The senior-engineer track. Take the failure modes from earlier in the stratum and design them out of existence. Every pattern here closes a real bug: idempotency, backfill-as-an-API, contract tests, dead-letter queues, and the migrations that move a team from cron to a real orchestrator without an outage.
Building Pipelines That Don't Page You
Idempotency as a design choice, backfill as a first-class API, small DAGs with sharp edges, contract tests at the source, partitioning for the reader, dead-letter queues for data, where to put state, and how to design for a one-engineer team. The build-side answer to every failure mode in this stratum.
8 ch · 1h 29m
1 freeMigrations and Deprecations
Migrating from cron to Airflow without an outage, deprecating a DAG nobody admits they own, moving between warehouses, splitting a monolith DAG, renaming a column safely, and a TheWorldShop case study on the migration that actually worked. The unglamorous half of platform engineering.
6 ch · 1h 9m
1 freePipeline Specs and Quality Checks
Gold pipelines start as documents, not DAGs. What data quality actually means (trust plus impact), Airbnb's nine-step Midas process and when it's overkill, the anatomy of a spec that survives review, and how to choose quality checks that fit dimensions vs facts, including guardrail metrics and the checks that catch security incidents.
4 ch · 1h 20m
1 freeRelated topics
Start Orchestration & Pipelines free
The first chapters of every course are free to read — no account needed.