The Building Blocks: A Data Engineer's Primitives

Maya, a staff engineer, sat in on a design review. The junior's pipeline worked, but it was bespoke: a hand-rolled queue here, a custom dedup table there, a cron job pretending to be a scheduler. She walked to the whiteboard and redrew the same system with five named boxes: a log, a table format, a stream processor, an orchestrator, a serving store. Suddenly the whole room could reason about it, argue about it, and spot the gap. Nothing changed except the vocabulary.

That's what this lesson gives you. System design is mostly composition: you're not inventing mechanisms, you're picking from a small set of well-understood primitives and wiring them together. Knowing the set, and the price tag on each, is most of the job. Here's the catalog.

The primitivesalmost every problem is a subset of this map

Sources

Operational DBEvents

Backbone

Log / Kafka

Storage & compute

Stream processorLakehouseBatch engine

Serving

OLAP engineOnline / KV storeDashboardsApps / ML

Cross-cuttingOrchestratorGovernance + catalog

Hover a primitive to see what it feeds and what it costs you.

Almost every problem in this offering is a subset of that diagram. Learn the pieces once and you can read any of them.

The log (Kafka)

What it is: a durable, append-only, replayable record of events that many producers write to and many consumers read from independently.

The log is the backbone of modern data systems. It decouples the thing producing data from the things consuming it. Producers don't know or care who reads; consumers read at their own pace and can rewind. Because it's durable and replayable, it doubles as a buffer that absorbs spikes and a source of truth you can reprocess from.

• Reach for it when: you have multiple consumers of the same stream, you need to absorb bursty load, or you want the ability to replay history. It's the default front door for event data and the transport for CDC.
• It costs you: an operational system to run (or a managed bill). Ordering is only guaranteed within a partition, not across the topic, so your partitioning key matters. And it's a pipe, not a store: it has a retention window, not infinite memory.
• Go deeper: Ingestion & Transport, the Kafka track (fundamentals → internals → operations).

Change data capture (Debezium)

What it is: a way to turn a database into a stream of its own changes by reading the database's transaction log, instead of repeatedly querying it.

CDC is how you get fresh data out of an operational database without hammering it or bolting "also write to the queue" onto every application (the dual-write problem, which corrupts data the moment one of the two writes fails). The DB already writes every change to its log for its own durability; CDC just tails that log and publishes it.

• Reach for it when: you need operational data fresh in minutes, the source is a real database, and you can't afford to run heavy queries against it or trust application-level dual writes.
• It costs you: care around the initial snapshot-plus-stream handoff, ordering and dedup on replay, and schema drift when the source changes a column out from under you. Deletes arrive as tombstones you have to handle deliberately.
• Go deeper: Ingestion & Transport, the CDC track (fundamentals and at scale).

Object storage + columnar files (Parquet on S3)

What it is: cheap, effectively infinite storage (S3, GCS, ADLS) holding column-oriented files (Parquet) that compress hard and let queries read only what they need.

This is the floor of the modern data stack. It's cheap, durable, and separates storage from compute so you can scale them independently. Columnar layout makes analytical scans fast and compression cheap.

• Reach for it when: you're storing large volumes for analytical access (scans and aggregations). Which is to say, almost always, as the landing zone and the lake.
• It costs you: high per-request latency (it's terrible for point lookups and tiny files), and no transactions or updates on its own. Raw object storage is a filesystem, not a database, which is exactly why the next primitive exists.
• Go deeper: Storage & File Formats, the Parquet deep dive.

Open table format + catalog (Iceberg / Delta)

What it is: a metadata layer over those Parquet files that adds database-like guarantees: ACID transactions, schema and partition evolution, time travel, and safe concurrent writes. A catalog tracks which files make up a table right now.

Table formats are what turn a pile of files into a table you can trust. They give you atomic commits (a reader never sees a half-written update), the ability to change schema or partitioning without rewriting everything, row-level updates and deletes (which raw Parquet can't do), and snapshots you can query as of a past point in time.

• Reach for it when: you want warehouse-like reliability on cheap lake storage. This is the heart of the "lakehouse". You need it the moment you have concurrent writers, updates/deletes (CDC merges, GDPR deletion), or evolving schemas.
• It costs you: a catalog to operate, and ongoing maintenance: compacting small files and expiring old snapshots, or your metadata and storage bloat. The copy-on-write vs merge-on-read choice trades read speed against write speed.
• Go deeper: Open Table Formats, the Iceberg and Delta tracks.

Stream processor (Flink, Kafka Streams)

What it is: an engine that transforms data in flight, while it moves, maintaining state across events: windows, joins, aggregations, with exactly-once guarantees.

When you need to act on data within seconds, and especially when the computation needs memory of past events (a running count, a session, a join against recent history), you need a stream processor. Flink is the heavyweight: real streaming, large managed state via RocksDB, exactly-once via checkpointing, event-time correctness via watermarks. Kafka Streams is a library you embed in a service for lighter jobs.

• Reach for it when: the freshness requirement is seconds and the logic is stateful: real-time aggregation, fraud scoring, sessionization, streaming joins.
• It costs you: real complexity. State has to be checkpointed and recovered, watermarks have to be tuned for your lateness, and an exactly-once sink needs transactional support. It's the most operationally demanding primitive on this list. Don't reach for it when batch would do.
• Go deeper: Ingestion & Transport, the stream processing track (Flink, Kafka Streams, and how to choose).

Batch compute engine (Spark)

What it is: an engine for large-scale, distributed transformation over data at rest: big joins, big shuffles, backfills, feature computation.

Batch is the workhorse, and it's underrated in interviews where everyone wants to look modern by streaming everything. Most analytics, most ML feature pipelines, and every "reprocess a year of history" job are batch. Spark shuffles and aggregates terabytes, and a batch job over Parquet files is usually cheaper and simpler than the streaming equivalent.

• Reach for it when: freshness in hours is fine, the data is at rest, or you're doing a heavy one-shot transform or backfill. When in doubt between batch and streaming, and the SLA allows it, batch is the cheaper, simpler default.
• It costs you: latency (it runs on a schedule, not continuously) and cluster cost while it runs. Big shuffles are where it gets slow and expensive, so partitioning and skew matter.
• Go deeper: Compute Engines, the Spark track (fundamentals, advanced, streaming).

OLAP serving engine (ClickHouse, Druid / Pinot, Trino)

What it is: a query engine built to answer analytical questions over big data fast: sub-second aggregations across billions of rows.

The lake is great for storing and transforming, but it's not built to power a dashboard a hundred analysts refresh all day. OLAP engines are. ClickHouse is a blazing columnar database for analytics and logs. Druid and Pinot specialize in real-time, sub-second dashboards over fresh data. Trino queries across many sources without copying data into one place (federation).

• Reach for it when: the serving access pattern is interactive analytical queries: dashboards, ad-hoc exploration, real-time metrics. Pick the engine by freshness and query shape.
• It costs you: another system to operate, and pre-aggregation choices that trade query flexibility for speed. Real-time OLAP engines are sensitive to cardinality and segment sizing.
• Go deeper: Query Engines & OLAP (ClickHouse, real-time OLAP, Trino, and designing OLAP systems).

Orchestrator (Airflow, Dagster)

What it is: the conductor that runs your pipelines in the right order, retries failures, handles dependencies, and lets you backfill history.

Once you have more than a couple of jobs, you need something that knows job B can't start until job A succeeds, that retries the flaky step, that can rerun last Tuesday because finance disputed a number. That's the orchestrator. It is not a compute engine; it tells the compute engines when and in what order to run.

• Reach for it when: you have dependencies between jobs, scheduled work, or any need to backfill and rerun. Which is every non-trivial batch system.
• It costs you: a scheduler to operate, and a discipline: tasks must be idempotent and carry their state, or retries and backfills corrupt data. The orchestrator gives you the ability to rerun; idempotency is what makes rerunning safe.
• Go deeper: Orchestration & Pipelines (Airflow internals, idempotency & state, backfills, SLA & lineage).

The serving and modeling layers

Two more primitives that decide how consumers actually experience your data:

• Online / key-value store. A low-latency store (Redis, DynamoDB, Cassandra) for point lookups: serving a feature to a model in milliseconds, a live counter, a leaderboard. The complement to the analytical lake. Reach for it when the access pattern is "get the value for this one key, right now." It costs you a second copy of the data and the job of keeping it fresh.
• Semantic / metrics layer + dimensional model. One agreed definition of a number. A star schema (facts and conformed dimensions), or a metrics layer that defines "revenue" once so every dashboard computes it the same way. Reach for it the moment two teams disagree about what a metric means. It costs you modeling discipline and governance. Go deeper: Semantic & Metrics Layer (dimensional modeling, the metrics layer, Data Vault and OBT).

The governance layer

What it is: the cross-cutting machinery for handling data responsibly: PII detection and masking, access control, deletion (the right to be forgotten), lineage, and a catalog so 10,000 tables stay discoverable.

Governance isn't a box at the end; it threads through every layer. In an interview it's the difference between a toy and a system that can run in a regulated company. Mention PII handling and a deletion story unprompted and you signal you've shipped real systems.

• Reach for it when: there's personal data (almost always), compliance in scope (EU, finance, health), or enough tables that nobody can find anything.
• It costs you: real work that doesn't show up on a dashboard: classification, masking pipelines, the awkward conflict between immutable time-travel snapshots and "delete this user everywhere."
• Go deeper: Privacy & Governance (PII foundations, masking & anonymization, PII at scale on Spark & Iceberg).

A decision table

When you know what you need, this is the fast path to the right primitive, and the thing to watch:

You now have the toolkit

Four lessons in, you have the method (what's being tested, the framework, the estimate) and the materials (these primitives). Everything after this is application: each problem page is a real design round, worked end to end, composing these blocks and defending the tradeoffs.

Start with the two free foundational problems, the Batch Ingestion + Warehouse Pipeline and the CDC Pipeline. They use almost every primitive here, and they're the cleanest way to watch the framework and the building blocks click together into a real system.