KRaft: How Kafka Replaced ZooKeeper With a Log

Apache Kafka spent fourteen years selling the industry one idea: if you want state to move reliably between machines, stop passing it around with RPCs that half-succeed, and stop asking a database "what's true right now?" Write an ordered, replicated, durable log. Let every consumer replay it. Let an offset be the single definition of how far along anyone is.

Kafka did not do this for itself.

Every Kafka cluster has exactly one active controller - a single elected node (a broker moonlighting as one in the ZooKeeper days, a dedicated quorum node in KRaft). It is the only component allowed to make decisions that are true for the cluster as a whole, and there is a short, specific list of them:

• Which topics and partitions exist, and what their replica assignments are.
• Who leads each partition - the single replica that accepts writes for it.
• Which replicas are in sync (the ISR), and therefore which ones are even eligible to be elected leader.
• Which brokers are alive, and what to do about the partitions on one that isn't.

That last one is the job that matters at 3am. A broker holding leadership for 4,000 partitions dies. Every producer writing to those partitions is now stalled. Nobody can fix this locally - a surviving replica cannot promote itself, because two replicas each deciding they're the leader is exactly how you lose data. Somebody has to pick 4,000 new leaders from the in-sync sets and tell the whole cluster. That somebody is the controller. It is a deliberate single point of coordination: leader election needs a decider, not a committee.

Which makes the controller's state the most important state in Kafka. Its in-memory picture is the cluster's idea of itself, and every broker's routing behavior is downstream of it.

And until Kafka 3.3, that state lived in an external, tree-shaped database called ZooKeeper, and reached the brokers as a stream of fire-and-hope RPCs that could half-arrive. When the controller itself died, its successor started from zero and rebuilt the entire picture out of that database before it could elect a single leader. On a large cluster that was minutes with no leader elections happening anywhere. Not degraded. Stopped.

So the system whose entire pitch was use a log managed its own most critical state with a database and a pile of best-effort RPCs.

KRaft is Kafka finally taking its own advice. The thesis is one sentence: the controller is the leader of a Kafka partition, and cluster metadata is that partition's log. Metadata stops being a tree you query and becomes a log you replay. Once that clicks, the rest of KRaft stops being a new distributed system to learn and becomes a system you already understand - because it has offsets, epochs, a high watermark, followers that fetch, and consumers that lag, exactly like every other Kafka topic you've ever debugged.

This post builds that idea from the log up, and then shows what it costs you.

The old world: metadata as a database

To see why KRaft is shaped the way it is, you need the failure mode it was built against.

In ZooKeeper mode, ZooKeeper was the store of record. Broker registrations, topic configs, partition assignments, and ISR membership all lived as znodes. A broker became the controller by winning a race to create an ephemeral /controller znode - the election was a lock, held outside Kafka entirely.

From there the loop was: read state out of ZooKeeper, hold it in memory, decide things, and then push the results to every other broker with LeaderAndIsr and UpdateMetadata requests. That push is where it fell apart, and KIP-500's own motivation section is blunt about it:

Read that again with the log-shaped part of your brain. A stream of state changes was being delivered with no ordering guarantee, no durable position, and no way for a receiver to say "I'm at change 41, send me 42 onward." Each RPC was fire-and-hope. A broker that missed one had no idea it had missed one. There was no offset, so there was no such thing as being behind - only being wrong.

Three consequences fell out of that design, and every one of them is a thing KRaft deletes rather than improves:

• Two sources of truth that drifted. ZooKeeper was the store of record, but the controller's in-memory state was what actually drove decisions, and the two disagreed constantly. When a leader changed its ISR in ZooKeeper, the controller often wouldn't learn about it for many seconds.
• Failover was a reload, not a resume. When the controller died, the next one started cold: read every topic, every partition, every ISR out of ZooKeeper, then push the world back out to every broker. That work is O(number of partitions), which is why controller failover on a big cluster was measured in minutes and why partition count was effectively capped by controller startup time.
• Ordinary operations paid the reload tax. Creating or deleting a single topic made the controller reload the full list of all topic names in the cluster.

The realization: metadata is a stream, not a tree

So here is the move. Every fact the controller manages is really a change over time: this topic was created, this partition's leader moved, this broker was fenced, this config was set. That's not a tree of nodes. That's an event log - and Kafka has an extremely good one.

Turn the model inside out:

• Metadata is the log. Not stored in a database and copied to a log: the log is the store of record. There is no second source of truth to drift from.
• The controller doesn't push. Brokers fetch, exactly like consumers, and track their position with an offset.
• Being behind is now a first-class, measurable state. "Broker 4 is at offset 9,912 and the leader is at 9,940" is a normal, observable, recoverable condition - where the old model would just have been silently wrong.

That last bullet is the one worth watching happen. Same slow broker, both protocols:

That single reframing is KRaft. Everything below is consequence.

The log itself: __cluster_metadata

Concretely, KRaft introduces one internal topic: __cluster_metadata, a single partition, replicated across the controller nodes. Its records are strongly typed schemas, and there are only 26 of them in the whole system. That's the entire vocabulary of Kafka's cluster state:

The state of your entire cluster is the result of replaying those records in order. Not a snapshot of a tree: a fold over a stream.

And now the punchline, straight from the class-level doc on QuorumController:

Controller election is not a separate mechanism. There is no /controller znode, no lock, no lease. The controller is the partition leader, so electing a controller is the same act as electing a leader for __cluster_metadata. Kafka already knew how to do that. One less distributed algorithm in the system.

Two more design choices in that same component are worth stealing for your own systems. QuorumController is single-threaded - one event-handler thread does almost everything, which the source justifies plainly: "This avoids the need for complex locking." It can afford to be single-threaded because it never does I/O against an external database; it just appends to a log. And its API is futures-based, where "the future associated with each operation will not be completed until the results of the operation have been made durable to the metadata log." Your createTopics() call returns when the record is committed. Durability is the API contract.

"A Kafkaesque version of the Raft protocol"

KRaft's consensus layer is Raft, but not textbook Raft, and the source code says so in the first line of KafkaRaftClient:

That sentence is the whole design. Split it in two:

Leader election is more or less pure Raft. Terms (called epochs), candidates, vote requests, majority wins, higher epoch always beats lower. Nothing exotic.

Replication is inverted. In textbook Raft, the leader pushes AppendEntries to followers. In KRaft, followers pull with Fetch - the same Fetch API a consumer uses. This is the single biggest departure, and it's a reuse decision: Kafka already had a hardened, high-throughput, batched, zero-copy replication path that had been running in production for a decade. Rewriting that as a push protocol for one internal topic would have been strictly worse.

But you don't get that reuse for free, and the protocol has a visible scar to prove it. Here's the API list from the same doc, with the giveaway:

• Vote - sent by a voter whose election timeout expired. Carries the last offset in its log, so electors can refuse to vote for a candidate that's behind. (This is Raft's safety rule: never elect a leader missing committed entries.)

• BeginQuorumEpoch - sent by a new leader to assert leadership. And the doc explains exactly why it has to exist:

  This is not needed in usual Raft because the leader can use an empty data push to achieve the same purpose. The Kafka Raft implementation, however, is driven by fetch requests from followers, so there must be a way to find the new leader after an election has completed.

  That's the cost of pulling. In push-Raft, a new leader announces itself by pushing an empty heartbeat. In pull-Raft, nobody is listening - followers are busy fetching from a leader that no longer exists. So KRaft adds one RPC whose entire job is to tap the followers on the shoulder and say fetch from me now.

• EndQuorumEpoch - sent by a leader resigning gracefully, telling voters to start an election immediately rather than wait out a timeout. This is why a clean controller shutdown fails over in milliseconds instead of seconds.

• Fetch - ordinary Kafka Fetch, plus the current leader and epoch piggybacked on the response, plus truncation detection folded into the same call.

• FetchSnapshot - for followers so far behind that the records they need are already gone. More on this below.

And the last clause of that thesis sentence - "we use Kafka's log reconciliation protocol to truncate the log to a common point following each leader election" - is the deepest reuse of all. When a new leader is elected, followers may hold records the new leader never had (uncommitted writes from the dead leader). Those records must go. KRaft resolves that with leader epochs and truncation: a follower asks "what's the end offset of epoch 7?", learns it diverged at offset 4,102, and truncates. That is bit-for-bit the same mechanism that stops log divergence in ordinary partition replication, which is covered in depth in the acks=all and the ISR post. KRaft didn't invent a reconciliation protocol. It called the one already in the building.

The state machine (and why brokers are boring)

QuorumState enforces the legal transitions, and there are six states. It's worth knowing them because the current-state metric reports exactly these names, and a cluster stuck in the wrong one is a specific diagnosis rather than a vague "election problem."

The transitions that matter:

• Follower → Prospective, on expiry of the fetch timeout. Note what triggers an election in a pull system: not a missing heartbeat from the leader, but the follower's own fetch failing to make progress. Liveness is measured by the thing doing the pulling.
• Prospective → Candidate, only after receiving a majority of granted PreVotes.
• Candidate → Leader, after a majority of real votes.
• Anything → Unattached, on learning of a higher epoch. Higher epoch always wins. This is Raft's core safety property and it's absolute.

That Prospective state is newer than most KRaft explainers, and it's there to fix a genuine Raft flaw. Without a pre-vote phase, a node that got partitioned away increments its epoch, gets rejected, times out, increments again, and so on - and the moment the partition heals it rejoins with a wildly higher epoch and forces the healthy leader to step down, even though nothing was wrong. A disruptive election caused entirely by a node that was never eligible to win. Pre-vote closes it: in Prospective, a node asks "would you vote for me?" without incrementing its epoch. If the answer is no - because the other voters are happily fetching from a live leader - nothing was disturbed. The isolated node never gets to punish the cluster for its own bad network.

Run the same partitioned node twice, once with pre-vote and once without, and watch the epoch counter rather than the states:

The final line of the QuorumState doc is the one that makes the architecture click:

Your brokers are observers. They fetch the metadata log, they apply it, they never vote, they can never become controller. A broker's relationship to cluster metadata is now identical to a consumer's relationship to a topic: find the leader, fetch, track a position, apply. That's why brokers scale to thousands without touching the quorum - adding an observer to a log costs the leader one more fetcher, and Kafka has always been good at fetchers.

Snapshots: a floor under an infinite log

The log model has one obvious problem. Cluster metadata is state, and the log is infinite. A cluster running for three years cannot make a new broker replay three years of PartitionChangeRecords to learn who leads partition 12 today.

Kafka's usual answer to "a log that represents state" is compaction - keep the latest record per key, drop the rest. That's how __consumer_offsets works. But KRaft doesn't compact __cluster_metadata. It snapshots it, and the difference is instructive.

Compaction is per-key and doesn't understand relationships between keys. Metadata is a web of cross-referencing records - a PartitionRecord is meaningless without the TopicRecord and the RegisterBrokerRecords it points at - and, crucially, a compacted log still needs a reader to fold it into state. A snapshot skips the fold: it's the materialized in-memory state, serialized. A new node doesn't replay it so much as load it.

The mechanics, straight from Snapshots.java and MetadataLogConfig:

• A snapshot is a file named for the offset and epoch it replaces, suffixed .checkpoint (built via a .checkpoint.part temp file, then atomically renamed - so a crash mid-write can never yield a torn snapshot that looks complete).
• Its name is a position: the snapshot at 00000000000000009900-0000000007.checkpoint means "the complete state as of offset 9,900 in epoch 7." Load it, then replay the log from 9,901. State and stream compose exactly.
• It's generated when either threshold trips: metadata.log.max.record.bytes.between.snapshots (bytes since the last snapshot) or metadata.log.max.snapshot.interval.ms (default 1 hour).
• Once state is captured in a snapshot, the log before it can be deleted, bounded by metadata.max.retention.bytes (default 100 MB) and metadata.max.retention.ms.

Which brings back the fifth API, FetchSnapshot:

A follower asks for offset 4,000; the leader deleted everything before 9,900 because it's in a snapshot. The leader can't serve 4,000 and won't pretend. Instead it replies "here's a snapshot ID," and the follower loads the snapshot and resumes from 9,901. This is the same shape as a consumer hitting OFFSET_OUT_OF_RANGE on an expired offset - except metadata can't just skip to the end, so KRaft hands it the state instead of an error.

Debugging KRaft

The payoff of "metadata is a log" is that your debugging instincts transfer. The questions are the same ones you ask about any topic: who's the leader, what's the epoch, how far behind is this consumer, is the state machine actually applying?

Who is the controller, and is the quorum healthy?

Start here, always:

kafka-metadata-quorum.sh --bootstrap-server localhost:9092 describe --status

That's the DescribeQuorum API. You get LeaderId, LeaderEpoch, the high watermark, and per-replica LogEndOffset for every voter and observer. Read it like consumer lag:

• A voter's LogEndOffset far below the leader's - that voter is lagging. If enough voters lag, you lose the majority that commits records, and all metadata writes stop cluster-wide even though every broker is up.
• LeaderEpoch climbing on its own - you have election churn. Each bump is a failover. A steadily rising epoch on a healthy-looking cluster means voters keep timing out on their fetches, usually from network trouble or a saturated controller disk.

The per-node metrics from KafkaRaftMetrics line up with it: current-state (the QuorumState names above), current-leader, current-epoch, high-watermark, and commit-latency-avg/commit-latency-max.

Is this broker's view of the cluster current?

This is the failure that has no ZooKeeper-mode equivalent, because in ZK mode there was no such thing as measurably-behind. Brokers are observers, and observers lag. The broker-metadata-metrics group is where you look:

A broker with a high last-applied-record-lag-ms is a broker acting on a stale view of the cluster: routing produce requests to a leader that moved, telling clients about partitions that were reassigned. It's up. It's healthy. It's wrong. And because the no-op record keeps the log ticking, this metric stays honest even on an idle cluster.

The two error counters deserve a harder look than they usually get. They should be zero, forever. Nonzero is not a transient blip to retry past - it means this broker hit a record it could not load or apply, so its in-memory image has now diverged from the log. That is precisely the divergent-state failure KIP-500 set out to eliminate, reappearing at a different layer. Every broker is running the same fold over the same ordered records, so a fold that fails on one broker and succeeds on others means the brokers no longer agree on reality. Alert on > 0 and treat it as a correctness incident, not a performance one.

Why did metadata writes stop when every broker is up?

Because broker health and quorum health are unrelated. __cluster_metadata commits by majority of voters, not by ISR - so a 3-node controller quorum tolerates exactly one failure. Lose two and the survivor cannot commit anything: no topic creation, no leader election, no config change, nothing. Meanwhile every broker keeps serving reads and writes off its last-known-good metadata image, so your dashboards look fine and your cluster is frozen.

This is the KRaft-specific instinct to build: the data plane and the metadata plane fail independently. Producers and consumers can hum along on stale metadata for a long time after the control plane has stopped. "Traffic looks normal" is not evidence that the quorum is alive.

The migration is over, and ZooKeeper's ghost is a record type

The timeline, for the record:

• 2019 - KIP-500 proposes replacing ZooKeeper with a self-managed metadata quorum.
• 3.3.1 - KRaft declared production-ready (KIP-833).
• 3.4 - 3.9 - the migration bridge: dual-write mode, a ZkMigrationState machine, and a supported path off ZooKeeper.
• 4.0 (March 18, 2025) - ZooKeeper support removed entirely. KRaft is the only mode.

So "debugging the ZK to KRaft migration" is now mostly a historical skill. The one live consequence: you cannot upgrade a ZooKeeper cluster straight to 4.0+. If you're still on ZooKeeper, the path is ZK 3.x → migrate to KRaft on 3.x → then upgrade to 4.0+. The bridge lives in the 3.x line, and 3.9 is the last stop.

Which leaves one last detail, and it's the best one in the codebase.

Grep the trunk source of Kafka 4.4 for "zookeeper" and the production code is effectively silent - one stray reference in a test tool. ZooKeeper is gone. But there is exactly one artifact left, sitting in the metadata record schemas next to TopicRecord and PartitionChangeRecord. It's ZkMigrationRecord.json, and its comment explains why it can never be deleted:

Sit with that. The last surviving trace of ZooKeeper inside Apache Kafka is a record type in the log that replaced it - retained not for ZooKeeper's sake, but because some cluster out there has a ZkMigrationStateRecord sitting at some offset in its __cluster_metadata log, and the log must remain replayable from the beginning. The database that Kafka depended on for fourteen years now exists only as an entry in Kafka's own log, kept alive by the very property that made the log the right answer all along: you don't get to rewrite history, you only get to append to it.

That's the real lesson of KRaft, and it outlives the ZooKeeper story. Kafka spent a decade selling everyone else on the log while managing its own state with a database and a pile of best-effort RPCs. The fix wasn't a better database or more reliable RPCs. It was noticing that the thing it had been building for everyone else was the thing it needed for itself.

KRaft is one of four mechanisms that decide how a Kafka cluster behaves under stress. For who owns which partition, see the two-part series on consumer group rebalancing. For what counts as durable - and where the leader epochs and log reconciliation in this post came from - see acks=all and the ISR. For what "exactly-once" actually means, see idempotence, transactions, and read-committed. For the foundations underneath all of them, that's the ingestion and transport curriculum.