Everyone wants exactly-once, most people think it's a checkbox, and almost nobody can say what the checkbox actually does - because there is no checkbox. "Exactly-once" in Kafka is three separate mechanisms that share a name: an idempotent producer that dedupes its own retries within a single session and partition; transactions that make writes to many partitions plus the consumer's offset commit atomic across sessions and fence zombie producers; and read-committed isolation that stops consumers from ever seeing uncommitted or aborted records. They solve different problems, fail in different ways, and - critically - are not all on by default. Idempotence is on; transactions and read-committed are opt-in, so the out-of-the-box producer is exactly-once in a much narrower sense than the phrase implies. This post builds all three from the protocol up - producer IDs, epochs, sequence numbers, the two-phase commit state machine, control markers, the Last Stable Offset - and then walks the failures each one throws, because knowing which of the three you're actually relying on is the whole game.
Almost everyone ships the same 'safe' config: replication.factor=3, acks=all, and a comfortable feeling that a committed message survives a broker dying. Then a broker dies and the message is gone anyway - and the config looks fine in the postmortem. The trap is that acks=all is a promise about the in-sync replica set (the ISR), not about your three replicas, and the ISR is a live set that shrinks whenever a follower lags. With the default min.insync.replicas=1, a shrunk ISR of just the leader still satisfies acks=all - so you're running acks=1 without a single config saying so. This post builds the real durability model from the replication protocol up: the high watermark that defines 'committed', min.insync.replicas as the actual floor, leader epochs that stop log divergence, unclean leader election as the availability-vs-safety switch, and the newest fix, Eligible Leader Replicas. Durability was never one setting - it's how five mechanisms interact.
Two years ago, Iceberg vs Delta Lake was a feature bake-off: who has row-level deletes, who has time travel, who has schema evolution. That comparison is dead. Both formats now have deletion vectors, row lineage, column mapping, and merge-on-read - they arrived at the same place from opposite directions, Iceberg through a version dial and Delta through table features. So the real decision has moved. It's not 'which format has feature X' anymore; it's which ecosystem you want to live in, which catalog you bet on, how much of your stack is Databricks, and how much the new interop layer lets you defer the choice entirely. This is that decision guide - what genuinely differs, what doesn't, and how to pick without regret.
Most engineers meet Delta Lake as 'the thing that gives Spark ACID transactions' and stop there. But underneath is one idea - an append-only transaction log of atomic commits - and a version story that looks nothing like Iceberg's clean dial. Delta grew by a ladder of protocol versions (writer versions 1 to 7, reader versions 1 to 3), bolting one capability onto each rung, until the ladder itself became the bottleneck. Writer 7 and reader 3 replaced it with table features: an a-la-carte set of named capabilities a table opts into individually. This is how the log works, how the ladder was climbed, and how deletion vectors, row tracking, column mapping, clustering, variant, type widening, and catalog-managed tables all hang off that pivot.
Part 1 built the machine: a consumer group is a set of partition leases watched by two clocks - a session/heartbeat clock asking 'is the process alive?' and a poll clock asking 'is it keeping up?' This part puts that model to work on the nine rebalancing failures that actually show up in production. Each one is a short story: the symptom you'd see on a dashboard, the log line that gives it away, the clock behind it, and the fix - which is almost never the first config the internet tells you to change. The recurring lesson is that the same visible symptom (lag spikes, endless rebalancing) has completely different causes, and the only way to tell them apart is to ask which clock fired before you touch a single setting.
Most engineers meet rebalancing as 'the thing that makes my lag spike when I deploy' and reach for a Stack Overflow answer that says 'increase max.poll.interval.ms.' Sometimes that works. Often it makes things worse, because the config they changed had nothing to do with the clock that actually fired. Kafka runs two independent liveness clocks over every consumer - a session/heartbeat clock that asks 'is the process alive?' and a poll clock that asks 'is it keeping up with its work?' - and every rebalance is one of those clocks expiring, a member intentionally joining or leaving, or a partition count change. This is how the whole machine works, from the coordinator handshake to the epoch-based reconciliation in the new protocol, built so that by the end you can name the clock behind any rebalance on sight.
Most engineers meet Iceberg as 'the format that gives you ACID on object storage' and stop there. But the format has a version dial - format-version 1, 2, 3, 4 - and each turn of it solved a real problem the last one left open. v1 made a table atomic on immutable files but could only append and overwrite whole files. v2 added row-level deletes, sequence numbers, and branches. v3 gave every row a stable identity, replaced position deletes with deletion vectors, and added richer types and encryption. v4 is the quiet refactor: relative paths, typed statistics, and the death of the file-system catalog. This is the continuity - what each version inherited, what it changed, and why.
The worst data modeling bugs don't throw. The query parses, the dashboard renders, the number has six significant digits and looks confident, and it's wrong. Revenue doubles because a join fanned out. Cash is off by $84M because a balance got summed across days. Two dashboards disagree by 3 points and a board meeting stalls. None of these are SQL typos - they're modeling decisions that were wrong from the start and only surface at reconciliation. This is a production field guide to the big ones: what the symptom looks like, why it happens, and the exact fix for each.
Most system design prep is built for backend engineers: design Twitter, design a URL shortener, talk about load balancers and read replicas. Then you walk into a data engineering loop and the prompt is design a near-real-time analytics pipeline, and none of the muscle memory transfers. Data system design turns on different hinges - data volume and velocity, batch versus stream, idempotency, backfills, schema evolution, partition skew, and freshness SLAs. The good news: there is a framework underneath it, and once you can run it live, every prompt starts to feel like the same five moves. This is that framework, with the exact questions to ask, the math to do out loud, and the follow-up questions interviewers use to find the edge of what you know...