Part 1 built the machine: a consumer group is a set of partition leases watched by two clocks - a session/heartbeat clock asking 'is the process alive?' and a poll clock asking 'is it keeping up?' This part puts that model to work on the nine rebalancing failures that actually show up in production. Each one is a short story: the symptom you'd see on a dashboard, the log line that gives it away, the clock behind it, and the fix - which is almost never the first config the internet tells you to change. The recurring lesson is that the same visible symptom (lag spikes, endless rebalancing) has completely different causes, and the only way to tell them apart is to ask which clock fired before you touch a single setting.
Most engineers meet rebalancing as 'the thing that makes my lag spike when I deploy' and reach for a Stack Overflow answer that says 'increase max.poll.interval.ms.' Sometimes that works. Often it makes things worse, because the config they changed had nothing to do with the clock that actually fired. Kafka runs two independent liveness clocks over every consumer - a session/heartbeat clock that asks 'is the process alive?' and a poll clock that asks 'is it keeping up with its work?' - and every rebalance is one of those clocks expiring, a member intentionally joining or leaving, or a partition count change. This is how the whole machine works, from the coordinator handshake to the epoch-based reconciliation in the new protocol, built so that by the end you can name the clock behind any rebalance on sight.
Most engineers meet Iceberg as 'the format that gives you ACID on object storage' and stop there. But the format has a version dial - format-version 1, 2, 3, 4 - and each turn of it solved a real problem the last one left open. v1 made a table atomic on immutable files but could only append and overwrite whole files. v2 added row-level deletes, sequence numbers, and branches. v3 gave every row a stable identity, replaced position deletes with deletion vectors, and added richer types and encryption. v4 is the quiet refactor: relative paths, typed statistics, and the death of the file-system catalog. This is the continuity - what each version inherited, what it changed, and why.
The worst data modeling bugs don't throw. The query parses, the dashboard renders, the number has six significant digits and looks confident, and it's wrong. Revenue doubles because a join fanned out. Cash is off by $84M because a balance got summed across days. Two dashboards disagree by 3 points and a board meeting stalls. None of these are SQL typos - they're modeling decisions that were wrong from the start and only surface at reconciliation. This is a production field guide to the big ones: what the symptom looks like, why it happens, and the exact fix for each.
Most system design prep is built for backend engineers: design Twitter, design a URL shortener, talk about load balancers and read replicas. Then you walk into a data engineering loop and the prompt is design a near-real-time analytics pipeline, and none of the muscle memory transfers. Data system design turns on different hinges - data volume and velocity, batch versus stream, idempotency, backfills, schema evolution, partition skew, and freshness SLAs. The good news: there is a framework underneath it, and once you can run it live, every prompt starts to feel like the same five moves. This is that framework, with the exact questions to ask, the math to do out loud, and the follow-up questions interviewers use to find the edge of what you know...
I didn't run a formal survey - nobody handed out a clipboard. But if you read enough postmortems, sit in enough on-call channels, and watch enough 'I wish someone had told me this' threads go by, the regrets of a hundred data engineers start to rhyme. The surprising part isn't that people wish they'd learned more tools. It's that the same depth gaps recur, almost word for word: the model that lied and never errored, the retry that double-charged a customer, the two files with identical rows that cost ten times apart. Here are those recurring lessons, grouped into the themes they actually fall into - each with a free place to feel it.
Every data engineer roadmap written before 2024 quietly assumed the bottleneck was writing the code. Learn Python and SQL, build a pipeline, ship it. That bottleneck is gone - AI writes most of it now, and writes it faster than you. So the 2026 roadmap can't be a checklist of tools; it has to be a map of the depth underneath them, the part that decides whether the AI-generated query is right and why the 3am pipeline failed. Here's that map, area by area, junior to senior, with where each layer actually bites - and where to practice it.
Every Spark ETL job grows the same crust over time: checkpoint wiring you copy between projects, incremental-ingest logic nobody wants to touch, a hand-maintained order of operations that breaks the moment someone adds a table. Spark Declarative Pipelines, the framework Databricks built as Delta Live Tables and then donated to Apache Spark, where it shipped in 4.1, tries to delete that crust. You declare the tables that should exist and what they contain; SDP figures out the order, the parallelism, the checkpoints, the retries. We went through the model piece by piece - flows, streaming tables, materialized views, the dataflow graph - to see what it genuinely handles for you and what it quietly hands back.
DuckDB has always been a single-process, embedded engine - extraordinary at what it does, and silently solitary. Quack puts a small network surface on top of that engine so two (or twenty) DuckDB processes can mount each other's catalogs and query each other's tables as if they were local. The defining choice is that it does this inside DuckDB, not around it: same wire format, same type system, same SQL. We stood up two servers, pointed them at each other, and went looking for where that one decision quietly decides everything else.