acks=all Is Not a Durability Setting: How Kafka Really Decides Your Data Is Safe

Here is a config almost every team ships and almost every team believes is safe:

replication.factor=3        # three copies on three brokers
acks=all                    # wait for "all" replicas before acking
min.insync.replicas=2       # ... if you were careful. Often left at the default.

The story you tell yourself is clean: a producer's write isn't acknowledged until all three brokers have it, so a message that got an ack survives any single broker dying. You've paid for three copies; you're getting three copies.

Then a broker dies during an incident, a message that was definitely acknowledged is definitely gone, and the config in the postmortem looks completely fine. Nobody changed acks. Nobody changed the replication factor. So what broke?

The answer is that acks=all does not mean "wait for all replicas." It means "wait for the current in-sync replica set" - and that set is a live thing that shrinks the instant a follower falls behind. Durability in Kafka was never a single setting. It's the interaction of five mechanisms - the ISR, the high watermark, min.insync.replicas, leader epochs, and unclean leader election - and if you only know one of them, the other four will eventually lose your data in a way that looks blameless. This post builds all five from the replication protocol up, in the order they bite.

The primer: what "committed" actually means

Start with the model, because the words "replica" and "committed" carry more weight than they look.

A partition has one leader replica and some follower replicas, spread across brokers. All reads and writes go to the leader. Followers don't get pushed to - they pull, each one continuously sending Fetch requests to the leader exactly like a consumer, copying records into their own log. That symmetry matters: a follower falling behind is the same event as a slow consumer falling behind, measured the same way.

Two offsets describe every replica's log:

• LEO (Log End Offset) - the offset just past the last record this replica has written locally. The leader's LEO jumps the moment a producer's batch lands; each follower's LEO trails as it fetches.
• HW (High Watermark) - the offset up to which records are committed. This is the load-bearing concept. A record below the HW is durable and visible; a record at or above the HW is written but not yet committed.

The gap between those two offsets is the entire subject of this post. Watch it open when a batch lands, and close only when the slowest in-sync follower catches up:

And the single rule that ties them together, straight out of the leader's maybeIncrementLeaderHW in Partition.scala:

That's the whole commit protocol in one sentence. A record is committed only once every in-sync replica has it (so the min of their LEOs has passed it). And here is the consumer-facing consequence people miss: consumers can only read up to the high watermark. A record your producer wrote but that hasn't reached the HW is invisible to consumers - not because it's hidden, but because it isn't safe yet. "Committed," "durable," and "visible to consumers" are the same line.

The ISR is a live set, not your replica list

Now the piece that turns a clean model into a trap. The "in-sync replica set" - the ISR - is not the same as "your replicas." It's the subset of replicas the leader currently considers caught up, and membership is constantly re-evaluated.

A follower is in the ISR only while it keeps fetching up to the leader's LEO within a time window: replica.lag.time.max.ms, default 30 seconds. Miss that window - because the follower's broker is GC-pausing, its disk is slow, the network is congested, or it's just restarting - and the leader shrinks the ISR, evicting that follower. When the follower catches back up, the leader expands the ISR to re-add it. (In KRaft, these changes are proposed to the controller via AlterPartition, so the cluster agrees on who's in-sync.)

So the ISR breathes. A healthy three-replica partition sits at ISR = 3, but under load, during a deploy, or on a bad-disk day it can drop to 2, or to 1 - just the leader - and then recover, all without anyone touching a config. That breathing is where durability leaks, because acks=all is defined against whatever the ISR happens to be right now.

Watch it happen. Nothing below is a config change - it is one slow follower, then another, and the same acks=all producer meaning something different at each step:

The default config gives you acks=1 in disguise

Watch the trap close with nothing but defaults. Modern Kafka ships:

Put acks=all (default) together with min.insync.replicas=1 (default) and trace it through a bad moment:

The producer in the right-hand case got a successful ack for a write that lived on exactly one broker. When that broker fails, a committed message is lost, and every config reads normal because the config was never the guarantee - the live ISR was, and it quietly dropped to one.

min.insync.replicas is the actual durability floor

This is why min.insync.replicas - not acks - is the real durability dial. acks=all decides who the leader waits for; min.insync.replicas decides how small the ISR is allowed to get before the leader refuses to pretend a write is safe. It's a floor, enforced at two points in the leader's code:

• Before the append, if the ISR is already below the floor, the write is rejected outright with NotEnoughReplicas.
• After appending locally but while waiting for replication, if the ISR has since shrunk below the floor, the producer gets NotEnoughReplicasAfterAppend.

Both are the leader saying "I cannot honor the durability you asked for, so I will fail your write instead of lying about it." That failure is a feature - it's the system refusing to accept data it can't keep safe. Set the floor to 1 and you disable that refusal entirely.

The canonical safe recipe falls straight out of this: replication.factor=3, min.insync.replicas=2, acks=all. Three copies, never acknowledge a write that isn't on at least two of them, and tolerate exactly one broker failing while staying both durable and available. Drop to min.insync.replicas=1 and you keep availability but lose the guarantee; raise it to 3 and a single lagging follower blocks all writes. Two is the sweet spot, and it is not the default - you have to set it.

Leader epochs: why "committed" survives a leader change

There's a deeper failure the model so far can't prevent: when the leader changes, how do we guarantee that a record everyone agreed was committed doesn't get un-committed? Followers can be slightly ahead of the HW (they've fetched records that aren't committed yet). If a follower with uncommitted extra records becomes leader, those records suddenly look committed; if a follower that was behind becomes leader, records that were committed can vanish. Early Kafka truncated logs using the high watermark alone and could genuinely diverge two logs during back-to-back leader changes.

The fix is leader epochs (KIP-101). Every leadership term gets a monotonically increasing epoch number, and each leader stamps the records it writes with its epoch, keeping a small leader-epoch-checkpoint mapping "epoch N began at offset X." Now, when a follower reconnects to a new leader, it doesn't blindly truncate to the HW. It asks the leader, via an OffsetsForLeaderEpoch request, "at what offset did your latest epoch that I also know about end?" - and truncates to exactly that divergence point. Two logs can no longer silently disagree about history; the epoch boundary is an authoritative fence for where one leader's truth ends and the next begins.

You mostly meet leader epochs through their error messages - FencedLeaderEpoch (you're talking to a replica about a stale epoch) and UnknownLeaderEpoch - and through the fact that log divergence, a real bug in 2016-era Kafka, simply doesn't happen anymore. It's the quiet mechanism that makes "committed" mean the same thing across every leader election.

Unclean leader election: the availability-vs-durability switch

Now the switch that most directly trades your data for uptime. Suppose every replica in the ISR is offline - all brokers holding an in-sync copy are down. The partition has no eligible leader. What should Kafka do?

Two choices, and Kafka makes you pick:

The default is false, and it should almost always stay there: it means Kafka would rather make a partition temporarily unavailable than silently lose data you were promised was safe. Flipping it to true is a deliberate statement that, for this topic, uptime matters more than never losing a record - a legitimate choice for some lossy telemetry, and a footgun for anything financial. The important thing is to know it's a choice you are making, not a default to leave unexamined.

The Last Replica Standing bug - and the fix, Eligible Leader Replicas

Here's the subtle part, the one even careful teams running RF=3, min.insync=2, acks=all, unclean=false didn't know they had. It's called Last Replica Standing, and it's why Kafka 4.x added a whole new mechanism.

Walk it through. Your ISR shrinks under load from {l, f1, f2} to {l, f1} to, finally, {l} - just the leader. With min.insync.replicas=2, new writes are now rejected (good - the floor is doing its job). But the already-committed records still live only on that one leader for the moment. Now that leader suffers an unclean shutdown - a power loss or kill -9 that loses un-flushed data from its disk. It restarts having lost some committed records. Because it was the last in-sync replica, it gets re-elected leader, and when f1 and f2 rejoin, they truncate their logs to match the leader - deleting the very records the leader lost. A local disk fault on one broker becomes global data loss of committed records. min.insync=2 didn't save you, because durability collapsed to one replica before the crash, and the survivors deferred to the amnesiac leader.

The fix, shipped as production in Kafka 4.1, is KIP-966: Eligible Leader Replicas (ELR). The core idea is to separate the replication quorum from the election quorum:

• The replication quorum is the ISR the leader knows - who acks writes, as before.
• The election quorum is the ISR plus the ELR, a new set the controller tracks of replicas that were kicked out of the ISR but are known to still hold all committed data (they were cleanly fenced, not data-lost).

The mechanics are tight and worth knowing:

The payoff, in the KIP's own framing: it provides min.insync.replicas - 1 tolerance to unclean-shutdown data-loss events, closing the gap where a single disk fault on the last in-sync replica could erase committed records everywhere. It's enabled via the eligible.leader.replicas.version feature and changes the semantics of min.insync.replicas slightly (it becomes a cluster-level invariant the controller enforces), which is why it shipped gated behind a metadata version rather than flipped on silently.

You don't have to configure ELR to benefit from understanding it: it's the clearest statement of what Kafka has always been trying to guarantee - that committed means "on at least min.insync.replicas brokers, always" - finally made true even across the nastiest crash.

The durability recipe, and how to read the symptoms

Put the five mechanisms together and durability stops being a mystery config and becomes a short checklist plus a set of metrics that tell you when a guarantee is eroding before it costs you.

The recipe for a topic you can't afford to lose:

# Topic
replication.factor=3
min.insync.replicas=2               # the real floor - NOT the default of 1
unclean.leader.election.enable=false # keep the default; durability over uptime
# Every producer to this topic
acks=all                            # default, but verify no producer overrides to 1
enable.idempotence=true             # default; also prevents duplicate/reordered retries
# Cluster (Kafka 4.1+)
eligible.leader.replicas.version=1  # close the Last Replica Standing gap

The symptoms to watch, and what each one is really telling you:

The instinct to build is that UnderMinIsr and NotEnoughReplicas are not failures to silence - they are the system telling you, honestly and in advance, that it can no longer keep the promise you asked for. The dangerous state is the opposite: acks=all writes sailing through while the ISR sits at 1, because min.insync.replicas=1 turned off the alarm. Silence isn't safety.

The throughline

Read Kafka's durability model back to front and it collapses to one correction of one wrong belief. The belief is that acks=all plus three replicas means three copies of every acknowledged write. The correction is that acks=all waits for the in-sync replica set, and that set is alive: it shrinks when followers lag, and with the default min.insync.replicas=1 it can shrink all the way to the leader while still acking your writes as though they were safe.

Everything else is the machinery that makes "committed" trustworthy despite that. The high watermark defines committed as "on every in-sync replica." min.insync.replicas is the floor that decides how small in-sync is allowed to get before the leader refuses to lie. Leader epochs keep committed meaning the same thing across elections. Unclean leader election is the explicit switch for when you'd rather have uptime than the guarantee. And Eligible Leader Replicas finally makes the floor a hard invariant even when the last replica crashes. Durability was never one setting - it's how those five interact, and the default settings optimize for availability, not for the safety most people think they're getting.

So the next time someone says "we're safe, we run acks=all," you know the only question that matters: what is min.insync.replicas, and is it really 2? If the answer is the default, you're one lagging follower away from acks=1 - and the config will look fine the whole way down.

If you want the layer underneath this - how partitions, offsets, delivery semantics, and the consumer side fit together - that's the ingestion and transport curriculum. And for the other half of the Kafka operational story, how the consumer side decides who owns which partition, see the two-part series on consumer group rebalancing - the same theme runs through both: the guarantee you rely on is defined against a live, changing set, and knowing which set is the whole game.