Design a Metrics Monitoring and Alerting Pipeline

Every system that runs in production needs a second system watching it. By now PhoenixWorld has a lot of the first kind: the batch loads, the CDC pipeline, the streaming ETL, the OLAP serving layer, all humming away in production. And when one of them silently stalls at 3am - a Kafka consumer wedged, a disk filling, latency creeping past the SLA - nobody finds out until a seller complains the next morning. The platform team needs the watcher: a system that ingests metrics from every service, stores them, and pages a human the moment something is actually wrong.

Noor runs reliability for the platform, and this is her problem to build. Her first sketch is the obvious one: have each service write a number to a database every few seconds, and query it for dashboards. Then the real shape arrives. There are thousands of instances, each emitting hundreds of metrics, each metric split across label dimensions like service and status code, which multiply out to millions of distinct time series. Alerts have to fire within a minute, but a single network partition must not generate five hundred pages. And the whole thing has to keep working precisely when the infrastructure it monitors is on fire.

The tension that shapes the design is that this is two systems fused into one. There is a time-series database - write-heavy, brutally sensitive to cardinality, downsampled as data ages - and there is an alerting engine that evaluates rules continuously, fires only on sustained conditions, and is clever about not crying wolf. Get the storage wrong and the monitoring falls over under its own write load. Get the alerting wrong and the on-call engineer learns to ignore it, which is the same as having no monitoring at all. This chapter builds both.

Step 1 - Understand the Problem and Establish Design Scope

"Monitor everything" hides the two decisions that matter: how metrics get from the services into the store, and what turns a stored number into a page. The clarifying questions pin those down.

Candidate: What are we collecting, and from how many sources?

Interviewer: System and application metrics from every service - CPU, memory, request rate, error rate, latency, queue depth, pipeline lag. Assume a few thousand instances, each exposing a few hundred metrics.

Candidate: What does a single metric look like?

Interviewer: A named time series with labels. Something like http_requests_total{service="api", instance="10.0.1.4", status="500"}, producing a numeric value at each point in time. The labels are the dimensions you slice by.

Candidate: Who initiates collection - do the services push their metrics to us, or do we go and fetch them?

Interviewer: That's a design decision, not a given. Tell me what you'd choose and why.

Candidate: Good, I'll come back to it, because it's the first big fork. How fast does an alert need to fire?

Interviewer: Within about a minute of a condition becoming true and staying true. And it must not fire on a one-second blip.

Candidate: Who consumes the stored metrics, besides the alerting?

Interviewer: Dashboards - engineers running ad-hoc queries over recent data during an incident - and capacity planning over months of history.

Candidate: How long do we keep it, and how exact does it need to be?

Interviewer: Recent data at full resolution for debugging, say a couple of weeks. Older data can be coarser - we only need trends. And a lost sample here and there is fine; this isn't billing. What's not fine is missing a real alert or paging the on-call fifty times for one incident.

Candidate: That last sentence is the whole quality bar. Noted.

Two forks define the design: the collection model, and the machinery that keeps alerting both timely and quiet.

Functional requirements

• Collect time-series metrics (name plus labels plus a value over time) from thousands of services and instances.
• Store them so dashboards and ad-hoc investigation can query recent and historical data.
• Evaluate alert rules continuously and fire when a condition holds for a sustained duration, not on a transient blip.
• Group, deduplicate, and route notifications so one incident is one page, not hundreds.
• Keep recent data at full resolution and older data downsampled to control cost.