Series
Kafka Internals
How Apache Kafka works underneath: consumer group rebalancing, durability and the ISR, exactly-once, KRaft, producer batching, share groups, log compaction, tiered storage, and what the log looks like on disk.
7 posts
How Kafka Consumer Group Rebalancing Actually Works: The Two Clocks (Part 1 of 2)
Most engineers meet rebalancing as 'the thing that makes my lag spike when I deploy' and reach for a Stack Overflow answer that says 'increase max.poll.interval.ms.' Sometimes that works. Often it makes things worse, because the config they changed had nothing to do with the clock that actually fired. Kafka runs two independent liveness clocks over every consumer - a session/heartbeat clock that asks 'is the process alive?' and a poll clock that asks 'is it keeping up with its work?' - and every rebalance is one of those clocks expiring, a member intentionally joining or leaving, or a partition count change. This is how the whole machine works, from the coordinator handshake to the epoch-based reconciliation in the new protocol, built so that by the end you can name the clock behind any rebalance on sight.
Read more →Debugging Kafka Consumer Group Rebalancing: Nine Failures and the Clock Behind Each (Part 2 of 2)
Part 1 built the machine: a consumer group is a set of partition leases watched by two clocks - a session/heartbeat clock asking 'is the process alive?' and a poll clock asking 'is it keeping up?' This part puts that model to work on the nine rebalancing failures that actually show up in production. Each one is a short story: the symptom you'd see on a dashboard, the log line that gives it away, the clock behind it, and the fix - which is almost never the first config the internet tells you to change. The recurring lesson is that the same visible symptom (lag spikes, endless rebalancing) has completely different causes, and the only way to tell them apart is to ask which clock fired before you touch a single setting.
Read more →acks=all Is Not a Durability Setting: How Kafka Really Decides Your Data Is Safe
Almost everyone ships the same 'safe' config: replication.factor=3, acks=all, and a comfortable feeling that a committed message survives a broker dying. Then a broker dies and the message is gone anyway - and the config looks fine in the postmortem. The trap is that acks=all is a promise about the in-sync replica set (the ISR), not about your three replicas, and the ISR is a live set that shrinks whenever a follower lags. With the default min.insync.replicas=1, a shrunk ISR of just the leader still satisfies acks=all - so you're running acks=1 without a single config saying so. This post builds the real durability model from the replication protocol up: the high watermark that defines 'committed', min.insync.replicas as the actual floor, leader epochs that stop log divergence, unclean leader election as the availability-vs-safety switch, and the newest fix, Eligible Leader Replicas. Durability was never one setting - it's how five mechanisms interact.
Read more →Exactly-Once in Kafka Is Three Mechanisms Wearing One Name
Everyone wants exactly-once, most people think it's a checkbox, and almost nobody can say what the checkbox actually does - because there is no checkbox. "Exactly-once" in Kafka is three separate mechanisms that share a name: an idempotent producer that dedupes its own retries within a single session and partition; transactions that make writes to many partitions plus the consumer's offset commit atomic across sessions and fence zombie producers; and read-committed isolation that stops consumers from ever seeing uncommitted or aborted records. They solve different problems, fail in different ways, and - critically - are not all on by default. Idempotence is on; transactions and read-committed are opt-in, so the out-of-the-box producer is exactly-once in a much narrower sense than the phrase implies. This post builds all three from the protocol up - producer IDs, epochs, sequence numbers, the two-phase commit state machine, control markers, the Last Stable Offset - and then walks the failures each one throws, because knowing which of the three you're actually relying on is the whole game.
Read more →A Kafka Transaction Is a State Machine Living in a Log You Never Read
Most explanations of Kafka transactions stop at 'atomic writes across partitions,' which is true and tells you almost nothing about what is happening. A transaction is not a buffer the broker holds until commit, and it is not a lock. Your records are appended to their partitions the instant you send them, interleaved with everyone else's, visible on disk. What makes them a transaction is a separate machine: a transaction coordinator running a two-phase-commit state machine whose state lives in an internal replicated log, and a single control marker it stamps into each partition at the end. The commit never touches your records. It writes one marker, and the reader does the rest. This post builds that machine from the source up: the transactional.id and coordinator selection, the protocol RPC by RPC, the full state enum including the epoch-fence state, control markers and coordinator epochs, the Last Stable Offset and aborted-transaction index that make it visible, and what KIP-890's Transaction V2 changed about all of it.
Read more →KRaft: How Kafka Replaced ZooKeeper With a Log
Kafka spent fourteen years telling everyone that a replicated, ordered log was the right way to move state between systems - and the whole time, it kept its own metadata in a ZooKeeper tree, pushed to brokers over RPCs that could half-arrive and leave the cluster in a divergent state. KRaft is the moment Kafka started eating its own dog food: the controller quorum is a Kafka topic, the active controller is just the leader of that topic's only partition, and every broker is a consumer of it. That one idea collapses a whole category of problems - failover stops being a reload and becomes a replay - but it also means metadata now has offsets, epochs, a high watermark, and lag, and you debug it with the same instincts you use for any other Kafka log. This post builds KRaft from the log up: why the source code calls it 'a Kafkaesque version of the Raft protocol,' why replication is driven by fetch requests, how snapshots put a floor under an infinite log, and what to look at when a controller election goes wrong.
Read more →Your Producer Is a Batching Engine, Not a send() Call
The single most useful thing you can learn about the Kafka producer is that send() is not a send. It's an append to an in-memory buffer, plus a promise. A background thread you never created decides when bytes actually hit a socket, and it decides using rules that have nothing to do with your call site. That gap is where every confusing producer symptom lives: the p99 that sits suspiciously close to a round number, the async API that mysteriously blocks, the BufferExhaustedException that isn't really about memory, and the retry that quietly reorders your log. This post takes the producer apart - RecordAccumulator, Sender, BufferPool, BuiltInPartitioner - and rebuilds it as what it actually is: a batching engine with a send()-shaped front door. Includes the Kafka 4.0 change that flipped linger.ms from 0 to 5 and what it says about the whole design.
Read more →