Series
Kafka Internals
How Apache Kafka works underneath: consumer group rebalancing, durability and the ISR, exactly-once, KRaft, producer batching, share groups, log compaction, tiered storage, and what the log looks like on disk.
3 posts
How Kafka Consumer Group Rebalancing Actually Works: The Two Clocks (Part 1 of 2)
Most engineers meet rebalancing as 'the thing that makes my lag spike when I deploy' and reach for a Stack Overflow answer that says 'increase max.poll.interval.ms.' Sometimes that works. Often it makes things worse, because the config they changed had nothing to do with the clock that actually fired. Kafka runs two independent liveness clocks over every consumer - a session/heartbeat clock that asks 'is the process alive?' and a poll clock that asks 'is it keeping up with its work?' - and every rebalance is one of those clocks expiring, a member intentionally joining or leaving, or a partition count change. This is how the whole machine works, from the coordinator handshake to the epoch-based reconciliation in the new protocol, built so that by the end you can name the clock behind any rebalance on sight.
Read more →Debugging Kafka Consumer Group Rebalancing: Nine Failures and the Clock Behind Each (Part 2 of 2)
Part 1 built the machine: a consumer group is a set of partition leases watched by two clocks - a session/heartbeat clock asking 'is the process alive?' and a poll clock asking 'is it keeping up?' This part puts that model to work on the nine rebalancing failures that actually show up in production. Each one is a short story: the symptom you'd see on a dashboard, the log line that gives it away, the clock behind it, and the fix - which is almost never the first config the internet tells you to change. The recurring lesson is that the same visible symptom (lag spikes, endless rebalancing) has completely different causes, and the only way to tell them apart is to ask which clock fired before you touch a single setting.
Read more →acks=all Is Not a Durability Setting: How Kafka Really Decides Your Data Is Safe
Almost everyone ships the same 'safe' config: replication.factor=3, acks=all, and a comfortable feeling that a committed message survives a broker dying. Then a broker dies and the message is gone anyway - and the config looks fine in the postmortem. The trap is that acks=all is a promise about the in-sync replica set (the ISR), not about your three replicas, and the ISR is a live set that shrinks whenever a follower lags. With the default min.insync.replicas=1, a shrunk ISR of just the leader still satisfies acks=all - so you're running acks=1 without a single config saying so. This post builds the real durability model from the replication protocol up: the high watermark that defines 'committed', min.insync.replicas as the actual floor, leader epochs that stop log divergence, unclean leader election as the availability-vs-safety switch, and the newest fix, Eligible Leader Replicas. Durability was never one setting - it's how five mechanisms interact.
Read more →