The DAG You Forgot
It's 11:20. Finance flagged a 5× jump on the orchestration cluster. SRE sees the scheduler CPU flatlined at 100% and DAG parse times north of 40 seconds — new tasks now schedule minutes late. Nobody remembers writing anything expensive, and the last deploy was three weeks ago. Two different things are quietly compounding at once, and you have to name both.
The incident
The orchestration cluster's cost is up 5× and the scheduler is CPU-bound, adding minutes of scheduling latency. There was no recent deploy. This is not one regression — it's two independent problems (a parse-time CPU tax and a task-row explosion) that add up to one bill and one latency symptom. Fixing only one leaves the other running.
Symptoms on the table
- Scheduler CPU pinned ~100%; dag_processing loop 40-90s (was ~2s)
- metadata DB task_instance table +190M rows in 30 days; DB CPU high
- Cluster cost 5× — mostly worker-hours + DB, not object storage
- A vendor_sync DAG runs hourly, owned by a team that offboarded last month
- No deploy in 3 weeks, no config change in the audit log
Systems on the board
The real components in play for this incident — the surface you investigate when the clock starts.
What you'll practice
This is a timed, hands-on incident in the Incident Response. You diagnose the symptom, trace it to a root cause across real components, and ship a fix before the clock runs out — the same loop you run on call, without the production blast radius.
Members-only challenge
Solve it in the Simulation Arcade.
The interactive workspace — live metrics, the component map, and the fix you ship — runs inside Petascale Labs. Sign in to start the clock.
Related topics
Browse the full Arcade
Every challenge maps to a stratum in the curriculum.