The 2 AM That Wasn't 2 AM
It's 08:10. The exec revenue dashboard opened board prep at 4.1× yesterday's real number. No task went red — the Airflow UI is a wall of green. But daily_revenue shows 26 runs that all fired between 01:58 and 03:40 last night, and fct_revenue has ballooned. Nobody triggered a backfill. Finance wants the real number before the 09:00 standup.
The incident
A once-a-day revenue DAG executed 26 times overnight and inflated the warehouse revenue figure ~4×. Every run reported success. There was no failure, no page, and no manual backfill — the pipeline did exactly what it was told to do.
Symptoms on the table
- daily_revenue: 26 successful runs stamped 2026-06-25 through 2026-07-20, all executed last night
- fct_revenue row count for the trailing 30 days up 4.1× vs the 90-day median
- Every run is green — zero failed tasks, zero retries, zero SLA misses
- No manual `airflow dags backfill` in the audit log
- Yesterday's deploy touched dags/daily_revenue.py
Systems on the board
The real components in play for this incident — the surface you investigate when the clock starts.
What you'll practice
This is a timed, hands-on incident in the Incident Response. You diagnose the symptom, trace it to a root cause across real components, and ship a fix before the clock runs out — the same loop you run on call, without the production blast radius.
Members-only challenge
Solve it in the Simulation Arcade.
The interactive workspace — live metrics, the component map, and the fix you ship — runs inside Petascale Labs. Sign in to start the clock.
Related topics
Browse the full Arcade
Every challenge maps to a stratum in the curriculum.