Design a Log Analytics Platform

At 2am an alert fires: checkout is throwing 500s. The on-call engineer has one question, and the whole company's recovery time hangs on how fast it gets answered. What is the error? Somewhere in the last ten minutes, across three hundred services and a billion log lines, is the one stack trace that explains it. A logging platform is the machine that turns "somewhere in a billion lines" into an answer in seconds.

PhoenixWorld's logs are currently a mess of files on disk, one pile per host, rotated away after a day. When something breaks, engineers SSH into boxes and grep, and by the time they've found the right host the log has already rolled over. Noor, who built the metrics-and-alerting platform two chapters ago, keeps getting pulled into incidents that metrics can tell you are happening but not why. Metrics say the error rate tripled. Logs say why. She's asked to build the second pillar of observability: a place where every log line from every service lands, stays queryable, and can be searched in seconds.

Her first instinct is the industry default: stand up Elasticsearch, ship everything to it, let its inverted index make every field searchable. Then she does the math on the storage bill and stops. Logs are almost pure write: billions of lines a day, and all but a handful are never read. Indexing every token of every line, so that the 0.01% you eventually search is fast, means paying to index the 99.99% you never touch. This chapter is about building a platform that stays fast to search without paying that tax.

Step 1 - Understand the Problem and Establish Design Scope

"Build a logging system" hides three different products: a live-tail debugger, a compliance archive, and an analytics platform. The clarifying questions pin down which one, and at what scale.

Candidate: Who queries this, and what does a typical query look like?

Interviewer: Mostly engineers during incidents. The dominant query is "show me the errors for service checkout in this five-minute window," then drilling into the lines around one of them. Occasionally someone greps for a substring like a specific order ID across a service. Rarely, someone runs an aggregation, like error counts per service over a day.

Candidate: How fresh does a log line need to be after it's emitted?

Interviewer: Seconds. If checkout is on fire right now, an engineer needs to see the lines from ten seconds ago. This is not a batch report.

Candidate: How much log volume are we talking about, and is it steady?

Interviewer: About three hundred services. Call it a couple hundred thousand lines a second on an average day. And it is emphatically not steady: when something breaks, every service in the blast radius starts logging errors, so volume spikes exactly when you need the system most.

Candidate: How long do we keep logs, and is that the same for everyone?

Interviewer: No, it varies. Debug logs are worthless after a week. Application logs, maybe thirty days. Audit and security logs, a year, for compliance. Retention is per-stream, and cost matters a lot on the long tail.

Candidate: Are logs structured or free text?

Interviewer: A mix. Newer services emit structured JSON with fields. Legacy services emit free-text lines. We have to handle both, and ideally give them a common shape.

Candidate: How correct does this have to be? Can we drop a line under load?

Interviewer: Logs are not a ledger. Losing a tiny fraction under extreme load is acceptable if it keeps the system up. We'd rather drop 0.1% of debug lines than have ingestion fall over and lose everything. That said, we shouldn't silently drop audit logs.

With that, the shape is clear: a write-heavy, cost-sensitive, near-real-time search system, not a transactional one.

Functional requirements

• Ingest logs from every service, both structured JSON and free-text lines, and give them a common queryable shape.
• Make a line queryable within seconds of emission.
• Support the three query shapes: filter by service and time window, substring search within a service, and aggregations over time.
• Enforce per-stream retention, from days to a year.
• Survive ingest spikes during incidents without falling over.