Poke Until It Dies
It's 09:00. The Airflow UI is a wall of yellow — ~600 tasks queued across four teams, almost nothing running. Workers show single-digit CPU. The scheduler heartbeat is healthy. Nobody deployed anything. It looks like the whole platform is broken and idle at the same time, and three teams are already in your DMs asking why their morning loads haven't started.
The incident
A shared multi-tenant Airflow cluster has deadlocked: hundreds of tasks are queued, near-zero are running, and workers are idle. No task has failed. The scheduler is fine. Capacity exists on paper but nothing can claim it.
Symptoms on the table
- ~600 tasks in 'queued', ~0 'running', across all four tenant teams
- Worker CPU < 10%, memory well under limit — the fleet is idle
- default_pool shows 128 / 128 slots used
- Team Growth's partner_ingest DAG has 400+ ExternalTaskSensor tasks in 'running'
- Scheduler heartbeat healthy; no failed tasks, no retries, no code deploy
Systems on the board
The real components in play for this incident — the surface you investigate when the clock starts.
What you'll practice
This is a timed, hands-on incident in the Incident Response. You diagnose the symptom, trace it to a root cause across real components, and ship a fix before the clock runs out — the same loop you run on call, without the production blast radius.
Interactive challenge
Solve it in the Simulation Arcade.
The interactive workspace — live metrics, the component map, and the fix you ship — runs inside Petascale Labs. Sign in to start the clock.
Related topics
Browse the full Arcade
Every challenge maps to a stratum in the curriculum.