Data Engineering System Design
Whiteboard the data platform with confidence.
Backend system design teaches you to scale a URL shortener. Data engineering interviews ask something harder: keep data correct under failure, at scale, on a budget. Learn the method, then work one self-contained problem per page.
The full method and 1 foundational problem are free.
core!
1
Design a Search Indexing Pipeline
Keep a 50-million-product search index fresh as the catalog changes in Postgres, without the dual-write trap, without a shared-category edit melting the pipeline, and without taking search down to reindex.
2the plan
SourcesIngestProcessStoreServe
3how to answer
1Scope2Sizing3Design4Deep dives5Tradeoffs6Wrap-up
Anatomy of a problem page · 6 sections
One method. Every problem.
Every problem walks the same six moves, budgeted to a 45-minute interview. Here it is on one worked example - designing a search indexing pipeline, section by section.
1
Section 15 min
Problem & Scope
Pin down what the pipeline must do, the freshness and correctness guarantees it owes, and what you would explicitly punt.
Requirements · search index
✓ 50M products, < 1 min fresh
✓ No dual-write to the index
out of scoperelevance tuning
the catch: fresh without a dual-write
2
Section 25 min
Back-of-the-Envelope
The numbers a DE estimates on the whiteboard: volume, bytes per doc, index size, throughput. One sizing decision falls out of the math.
Sizing
products50 M
doc size~2 KB
index~100 GB
→ shards10
10 shards keeps each ~10 GB, one cluster
3
Section 310 min
High-Level Design
The architecture in one diagram, then a walk of the data's journey from source to serving. Name the components, not just the vendors.
Data flow
PostgresCDCIndexerSearchAPI
CDC is the spine - the app never writes the index
4
Section 415 min
Deep Dives
Three to five focused sub-sections on the parts that actually get interrogated. Here: keeping Postgres the single source of truth, no dual-write.
No dual-write
write commit to Postgres only
CDC capture the row change
index idempotent upsert of doc
Postgres stays the single source of truth
5
Section 55 min
Bottlenecks & Tradeoffs
Where it breaks at 10x, and the decisions with no clean answer: a shared-category edit fanning out to millions of docs, freshness vs cost.
Tradeoff
Full reindex
+ simple
- slow, costly
Incremental pick
+ fresh, cheap
- fan-out risk
incremental live, full reindex to backfill
6
Section 65 min
Interview Wrap-Up
What a strong candidate says: the talking points that signal seniority, the follow-ups an interviewer pushes on, and where to go deeper.
What a strong candidate says
“Drive the index off CDC, never a dual-write, so Postgres stays the source of truth - then reindex into a new alias and swap, so search never goes down.”signals seniority
The catalog
Start with the method. Then work the problems.
The method and 1 foundational problems are free. Everything else is included with your subscription.
The Methodwhat it tests · the framework · the math · the building blocks
01What Data Engineering System Design Actually TestsA data engineering system design round is not a backend round in disguise. You're judged on whether data stays correct while it moves, retries, and gets reprocessed. Here's the real rubric.02The Design Framework: A Spine for Any Data ProblemA repeatable, eight-step spine for attacking any data engineering system design problem, plus the two axes that decide every architecture and how to spend the 45 minutes.03Back-of-the-Envelope for DataHow to estimate volume, storage, throughput, and cost on a whiteboard without a calculator, and how the one number that matters decides your whole architecture.04The Building Blocks: A Data Engineer's PrimitivesThe dozen primitives you compose into any data system: the log, CDC, columnar storage, table formats, stream and batch engines, OLAP, orchestration, and the governance layer, with the honest cost of each.
Cheat Sheetsthe numbers and the mistakes · free · skim before the interview
01Key DE Numbers: The Back-of-Envelope Cheat SheetThe latency, throughput, sizing, and cost numbers every data engineer should have memorized for a system design interview. One dense reference page: the latency ladder, Kafka and disk throughput, Parquet sizing, cloud cost anchors, and the estimation shortcuts that turn a vague prompt into a defensible number.02Top 15 Data Engineering System Design MistakesThe fifteen mistakes that sink data engineering system design interviews, and the one-line fix for each: polling instead of CDC, no idempotency, unbounded state, forgetting backfills, ignoring skew, dual writes, and more. A pre-interview checklist of the traps interviewers are actively probing for.
Foundational
01Design a Batch Ingestion + Warehouse PipelineTurn an operational Postgres database into a dimensional warehouse that answers daily revenue, orders, and cohort questions. A full, four-step batch pipeline design: scope and estimation, the raw to marts layering, then a deep dive on incremental extract, idempotent loads, the star schema, SCD Type 2, orchestration, backfills, and reconciliation.02Design a CDC PipelineReplicate an operational database into the lakehouse within minutes, without hammering the source or double-counting on replay. A full, four-step change-data-capture design: scope, high-level design, deep dive, and wrap-up.03Design a Clickstream / Event Analytics PipelineTurn a firehose of web and mobile events into funnels, sessions, and near-real-time dashboards analysts trust. A full, four-step event analytics design: scope and estimation, the collector and event contract, then a deep dive on event-time vs processing-time, sessionization, dedup and bot filtering, and the Lambda vs Kappa decision.04Design a Metrics Monitoring and Alerting PipelineIngest time-series metrics from thousands of services, store them, and page a human when something is wrong - within a minute, without drowning the on-call in false alarms. A full four-step monitoring design: scope and estimation, the metric contract and collection model, then a deep dive on pull vs push, the time-series store, cardinality, and the alerting engine.
Core
01Design an Ad Click Event Aggregation SystemCount ad clicks per campaign per minute, at very high volume, exactly once, for billing. The canonical DE interview question, worked end to end: scope and estimation, the speed-plus-batch architecture, then a deep dive on windowing, exactly-once vs idempotency, deduplication, hot-campaign skew, and the reconciliation batch that makes the invoice correct.02Design a Real-Time Leaderboard and Counting ServiceTop-selling products and top buyers, updating live during a flash sale, over millions of entities, without melting a database or holding gigabytes of exact state. A full four-step design: scope and estimation, the serving and aggregation model, then a deep dive on sorted sets, exact vs approximate counting, top-K sketches, HyperLogLog, and rebuild-from-log.03Design a LakehouseOne storage layer that serves both BI and ML, with ACID and time travel, on cheap object storage that offers none of it. The lakehouse worked end to end: scope and estimation, the Iceberg metadata tree and catalog, then a deep dive on atomic commits and optimistic concurrency, copy-on-write vs merge-on-read, compaction and the small-file problem, snapshot expiry and time travel, hidden partitioning, and schema evolution.04Design a Streaming ETL Pipeline with Exactly-OnceEnrich a thin order stream into one canonical, clean stream that many teams can trust, and keep it correct under failure. A full four-step streaming-ETL design: scope and estimation, the enrichment contract and high-level design, then a deep dive on stream-table joins, point-in-time correctness, checkpointing, and exactly-once chosen per sink.05Design a Near-Real-Time OLAP Serving LayerSub-second dashboards over tens of billions of rows that are seconds-fresh, for thousands of concurrent sellers. A full four-step real-time OLAP design: scope and estimation, the segment and query-contract design, then a deep dive on the real-time to historical handoff, one-big-table vs joins, ingestion-time rollup, indexing, cardinality, and broker routing.06Design a Log Analytics PlatformCentralize logs from every service so an on-call engineer can find the one error line in a billion, within seconds, without the storage bill eating the budget. A full, four-step design: scope and estimation, the log-event contract and columnar data model, then a deep dive on parsing, the sort-key-as-index insight, hot/warm/cold tiers, retention, query patterns, cost control, scaling, and the cardinality bomb.07Design a Search Indexing PipelineKeep a 50-million-product search index fresh as the catalog changes in Postgres, without the dual-write trap, without a shared-category edit melting the pipeline, and without taking search down to reindex. A full, four-step design: scope and estimation, the search-document contract and three ways to denormalize, then a deep dive on the stream join, fan-out on write, out-of-order safety, and the blue/green reindex.08Design a Data Quality and Observability SystemCatch bad data before the CEO sees a wrong number, across thousands of tables, without paging everyone for every blip. A full, four-step design: scope and estimation, the check taxonomy and where the quality gate belongs, then a deep dive on statistical anomaly detection, write-audit-publish quarantine, lineage-aware alerting, and the difference between a green pipeline and a correct table.09Design a Federated Query and Lakehouse Access LayerLet analysts join the lakehouse, the warehouse, and a live Postgres replica in one SQL query without copying everything into one place first, and without a runaway query melting production. A full, four-step design: scope and estimation, the engine/storage split and where to draw the federate-vs-centralize line, then a deep dive on connectors and pushdown, the accidental-ETL trap, cross-source join planning, source protection, and hybrid materialization.10Design a Feature StoreServe the same feature to a model at training time and at inference time, without leaking the future into training data or drifting between the two paths. A full feature-store design (Uber Michelangelo style): scope, the offline and online stores, and a deep dive on point-in-time correctness, training-serving skew, freshness, and backfills.11Design Netflix's Recommendation PipelineTurn billions of plays and impressions a day into a labeled training set and a precomputed row of recommendations for 250 million users, when a title someone didn't watch might mean 'disliked' or just 'never saw it'. A recommendation data pipeline design: scope, the offline/nearline/online layers, and a deep dive on training data from implicit feedback.
Walk in ready to design the platform.
One method, drilled across every problem until it's a reflex. Free to start.
Learn the method