Design a Data Quality and Observability System

On Monday morning the CEO opens the revenue dashboard and the number is down forty percent. Nothing is broken. Every pipeline ran, every job is green, every server was healthy all weekend. And the number is still wrong. Upstream, a currency field quietly started arriving as null on Saturday, a sum silently dropped the rows it couldn't convert, and the dashboard faithfully rendered a garbage total. By the time a human notices, three teams have already made decisions on it.

Noor built PhoenixWorld's infrastructure observability: the metrics platform that watches error rates and latency, and the logging platform that answers why. Both told her, all weekend, that everything was fine. And they were right, about the infrastructure. The pipelines were healthy. The data was not. That gap is the whole problem. A system can be perfectly observable at the infrastructure layer and completely blind to whether the numbers it's producing are true.

Noor's realization is that she needs a third pillar, pointed not at the machines but at the data itself. Metrics watch the pipeline; this watches the tables the pipeline produces. And the hardest part isn't detecting a bad number, it's catching it before anyone reads it, without crying wolf so often that people mute the alerts, the way they eventually mute every noisy pager. This chapter builds that system.

Step 1 - Understand the Problem and Establish Design Scope

"Monitor data quality" spans everything from a single dbt test to a company-wide trust platform. The clarifying questions pin down the scope and the scale.

Candidate: What are we protecting, and for whom?

Interviewer: The tables that people and models actually consume: the curated, gold-layer tables behind dashboards, exec reporting, and ML features. The goal is to catch bad data before it reaches a consumer, not to audit every intermediate staging table.

Candidate: What counts as "bad data"?

Interviewer: Several things. The table didn't refresh on time (stale). The row count is wildly off (a load half-failed). A column's type or presence changed (schema drift). The values look wrong (null rate spiked, a distribution shifted, revenue went negative). We want all of those covered.

Candidate: At what scale? How many tables and pipelines?

Interviewer: Thousands of tables, hundreds of pipelines, on a lakehouse. You cannot hand-write a bespoke check for every column of every table. Coverage has to be mostly automatic, with room for teams to add specific business rules.

Candidate: When a check fails, can we stop the bad data from being published, or only alert after the fact?

Interviewer: Ideally stop it. Detecting bad data after the CEO has seen it is the failure we're trying to avoid. But we also can't halt every pipeline in the company every time one column looks slightly off.

Candidate: Who gets alerted, and how sensitive is the org to false alarms?

Interviewer: Very sensitive. The last tool got muted because it paged everyone for everything. Alerts must go to the owner of the broken table, not to forty downstream teams, and a check people don't trust is worse than no check.

Candidate: Batch, streaming, or both?

Interviewer: Focus on the batch lakehouse tables that feed reporting. Streaming quality is real but a harder edge; keep it out of the core.

With that, the shape is clear: broad automatic coverage, a gate that stops bad data before consumers read it, and alerting disciplined enough that people still trust it a year from now.

Functional requirements

• Run quality checks on gold tables as they're produced: freshness, volume, schema, distribution/null-rate, and custom business rules.
• Detect freshness and volume anomalies against historical baselines automatically, without a human setting every threshold.
• Gate publication: quarantine a bad partition or table so consumers keep reading the last good version instead of the bad one.
• Route alerts to the owning team by lineage, and suppress the downstream cascade from a single upstream cause.
• Make the distinction between pipeline health and data health first-class.