What Data Engineering System Design Actually Tests
Dev booked a "system design" round for a senior data engineering role and prepped the way he'd prep for a backend one. Load balancer, app servers, a cache, a sharded database. Forty minutes of "design a URL shortener" muscle memory. The interviewer drew an arrow out of a Postgres box and asked one question: "This orders table double-sends a row when the producer retries. Where in your design does that get caught?"
Dev didn't have an answer, because his design didn't have a place for it. The round wasn't about moving a request in and out. It was about moving data, and keeping it correct while it moved.
That gap is what this whole section closes. Before we design a single pipeline, you need to know what the interviewer is actually grading.
Data design runs the other direction
Backend system design starts at the user and works backward. What does the user need? What API serves it? What service backs the API? What database backs the service? You optimize the request you can see: p99 latency, cache hit rate, connection pools.
Data engineering design starts at the source and works forward. What data exists? How do we get it out? How do we transform it? How do we serve it to everyone who needs it, in the shape they need? Nobody is waiting on a single request. The thing you're protecting is the data itself, over time, across retries and replays and schema changes.
Same diagramming skill, opposite starting point. If you draw a load balancer first in a data round, you've already told the interviewer you think this is a backend problem.
"Scale" means something different here
When a backend interviewer says "now scale it", they usually mean more requests per second. More QPS, more concurrent users, more connections.
When a data interviewer says "now scale it", they mean one of these:
- Volume went up. 10 GB/day became 2 TB/day. Your nightly job that finished in an hour now doesn't finish at all.
- Sources went up. 5 tables became 500 tables, each with its own schema and owner.
- Schemas drift. A source team renamed a column and didn't tell you. Half your pipeline is now reading nulls.
- Regions multiply. The business opened in the EU, and now there's data residency law in the path.
A good answer survives all four without a rewrite. That's the bar. The interviewer is probing whether your design bends or snaps when the shape of the data changes underneath it.
The priorities flip
Backend systems optimize for the request. Data systems optimize for the throughput and the rewrite. The defaults invert:
| Backend instinct | Data engineering instinct |
|---|---|
| Low latency per request | High throughput per batch or stream |
| Random reads and updates | Append-heavy writes, scans over ranges |
| Point lookups (by primary key) | Analytical scans (aggregate millions of rows) |
| Normalize to avoid anomalies | Denormalize to avoid joins at read time |
| Strong consistency, now | Eventual consistency, plus a way to reconcile |
None of these are absolute. The point is that if you reach for a backend default without saying why, you'll design the wrong system. A senior candidate names the access pattern first, then picks the storage to match it.
The five questions every answer has to survive
Here's the rubric. Whatever you design, the interviewer is silently checking whether it answers these five. Miss one and a "working" design quietly becomes a broken one in production.
- Data contracts and schema evolution. What is the shape of the input, who owns it, and what happens the day they change it? A design with no contract is a design that pages you on someone else's deploy.
- Idempotency and exactly-once. The source will double-send. The job will retry. Can the same record land twice without corrupting a count or a balance? If reprocessing the same hour twice changes your numbers, you don't have a pipeline, you have a slot machine.
- Late and out-of-order data. Events don't arrive in the order they happened. A mobile event from 9:02 shows up at 9:47 because the phone was in a tunnel. How late is too late, and what do you do with the straggler?
- Backfills and reprocessing. You will find a bug. You will need to recompute a year of history without taking production down or double-counting. If your design can only run forward, it's already obsolete.
- Cost and SLA. How fresh does this need to be, and what does that freshness cost? "Real time" is a price, not a feature. Naming the SLA and the bill is what separates an engineer from a wishlist.
Backend design optimizes the request you can see. Data design optimizes the replay you can't: the reprocessing job that runs at 3am six months from now, when someone recomputes a quarter of history and needs the numbers to come out identical. Design for that run, not just the happy first one.
The interviewer's hidden scorecard
Two candidates can draw the same boxes. One gets the offer. The difference is almost never the tools. It's the order and the honesty of the reasoning. Strong signals, in roughly the order they earn points:
- You clarify before you draw. You ask "how fresh, how correct, how big, what's the budget" before naming a single technology. Drawing Kafka in the first thirty seconds is a tell that you're pattern-matching, not designing.
- You name a delivery guarantee out loud. At-least-once with idempotent writes? Exactly-once with checkpointing? Saying which, and why, signals you've run this in production.
- You talk about failure modes unprompted. Backpressure, schema drift, data skew, a poison record, a replay. Mid-level candidates design the happy path. Senior candidates design the 3am path.
- You put a number on it. Even a rough one. "About 50 GB a day raw, call it 10 in Parquet, so streaming the reconciliation is affordable here." Estimation turns hand-waving into engineering.
- You state the tradeoff you're accepting. Every choice costs something. Saying "I'm taking eventual consistency here, and reconciling in a nightly batch" is worth more than any diagram.
Notice what's missing: nobody is grading whether you memorized the internals of a specific engine. They're grading whether you reason like someone who's been on call.
How to use this section
The next three lessons give you the toolkit:
- The Design Framework is the repeatable spine you walk through in every round, so you never freeze on "where do I start".
- Back-of-the-Envelope for Data is how you put that number on it without a calculator.
- The Building Blocks is the vocabulary: the dozen primitives you compose into any data system, and what each one costs you.
After that, every problem page is one of these rounds, worked end to end. Most follow our running company PhoenixWorld; a few are set at a real-world system like Uber, Netflix, or Stripe, where the problem is best known by that name. By the time you reach the capstone, you'll have designed an entire data platform one subsystem at a time.
Heads up on style: across this section, "correct" almost always beats "fast". When a tradeoff is genuinely close, we'll say so. When it isn't, we'll tell you which way to lean and why.