The $516k Gap
Friday, 08:47. Finance Slack just pinged: the Q2 revenue tile on the CFO Looker dashboard reads $4.21M, but the ERP reconciliation says $3.69M. Six days of refunds appear to have vanished somewhere between Postgres and the boardroom. The CEO walks into the board room at 09:00.
The incident
It's Friday, 08:47, and the CEO walks into the boardroom at 09:00. Finance just pinged: the Q2 revenue tile on the CFO's Looker dashboard reads $4.21M, but the ERP reconciliation — the system of record — says $3.69M. That's a $516k overstatement, and when someone pulled the history, the dashboard turns out to have been wrong for six days, drifting by about $86k a day. Nothing looks broken: the CDC connector is green, dbt is passing, the dashboard is fresh, and not one alert or PagerDuty page fired in those six days. The number is simply, quietly wrong somewhere between Postgres and the boardroom — and the board sees it in 13 minutes.
Symptoms on the table
- Looker revenue tile: $4.21M · ERP system of record: $3.69M (Δ +$516k overstated)
- fct_orders row count is growing normally — no freshness anomalies in the data pipeline
- Debezium connector lag: 0 ms · status RUNNING · all green
- dbt run history: 144 consecutive successes · every test passing
- Zero PagerDuty alerts, zero Slack #data-alerts notifications in the six days the bug has existed
- The orders team shipped a `refunded_at` column two months ago — the only recent schema change
Systems on the board
The real components in play for this incident — the surface you investigate when the clock starts.
What you'll practice
This is a timed, hands-on incident in the Incident Response. You diagnose the symptom, trace it to a root cause across real components, and ship a fix before the clock runs out — the same loop you run on call, without the production blast radius.
Members-only challenge
Solve it in the Simulation Arcade.
The interactive workspace — live metrics, the component map, and the fix you ship — runs inside Petascale Labs. Sign in to start the clock.
Related topics
Browse the full Arcade
Every challenge maps to a stratum in the curriculum.