The Green DAG That Lied
It's 14:30. No red tasks. No failed runs. No SLA misses. But the churn model's precision has slid for a week, and someone finally traced it to a feature table that's been landing ~50% populated every single day. The pipeline swears it succeeded — nine days running. Retrying today's run changed nothing.
The incident
A daily feature-export pipeline has reported success for 9 straight days while writing only about half the expected rows. The gap is stable, silent, and self-perpetuating — reruns succeed and fix nothing. A downstream model degraded before anyone noticed the data was incomplete.
Symptoms on the table
- feature_export: 9 consecutive 'success', exit code 0, no retries
- feature_table daily row count ~48-55% of the 90-day expected, every day
- The shortfall is stable, not random — always roughly half
- watermark table advanced past all 9 days as if they were complete
- Churn model AUC down from 0.88 to 0.79 over the same window
Systems on the board
The real components in play for this incident — the surface you investigate when the clock starts.
What you'll practice
This is a timed, hands-on incident in the Incident Response. You diagnose the symptom, trace it to a root cause across real components, and ship a fix before the clock runs out — the same loop you run on call, without the production blast radius.
Members-only challenge
Solve it in the Simulation Arcade.
The interactive workspace — live metrics, the component map, and the fix you ship — runs inside Petascale Labs. Sign in to start the clock.
Related topics
Browse the full Arcade
Every challenge maps to a stratum in the curriculum.