Exactly-Once in Kafka Is Three Mechanisms Wearing One Name

Start with the loop that everyone actually writes - consume a record, do some work, produce a result, commit the offset:

while (true) {
  var records = consumer.poll(...);
  for (var r : records) {
    var result = transform(r);
    producer.send(new ProducerRecord<>("output", result)); // (1) write
  }
  consumer.commitSync();                                     // (2) commit offset
}

There are two independent ways this loop produces wrong answers, and they are different bugs. First: producer.send fails at the network layer, the client retries, and the broker - having actually received the first attempt - writes the record twice. Second: the process crashes after (1) but before (2), so on restart it re-reads the same input and produces the result again. One is a duplicate from a retry; the other is a duplicate from reprocessing. They have nothing to do with each other, and "turn on exactly-once" is not a fix for either until you know which machine addresses which.

Because that's the thing nobody says plainly: there is no "exactly-once" switch. What Kafka gives you is three separate mechanisms that share the name, each solving a different one of these problems:

1. The idempotent producer - kills the first bug (duplicate from a retry), within a single producer session and partition.
2. Transactions - kill the second bug (duplicate from reprocessing) by making the writes and the offset commit one atomic unit, and by fencing zombie producers across restarts.
3. Read-committed isolation - makes the first two visible, by stopping consumers from ever reading uncommitted or aborted records.

They fail differently, they're configured differently, and - the part that trips up almost everyone - they are not all on by default. This post builds all three from the protocol up, then walks the failures each one throws, because the only way to debug "we have exactly-once but we're seeing duplicates" is to know which of the three you were actually relying on.

Mechanism 1: the idempotent producer

The narrowest and most misunderstood, because it's the one that's on by default and so people credit it with far more than it does.

The problem it solves is precise: a producer sends a batch, the ack is lost to a network blip, the producer retries, and the broker appends the same records a second time. Without help, "at least once" delivery means retries create duplicates. The idempotent producer removes exactly this duplication and nothing else.

The mechanism is a per-message dedup key. When idempotence is on, the producer gets a Producer ID (PID) and an epoch from the broker via InitProducerId, and it stamps every batch with three things:

• the PID (which producer),
• the epoch (which incarnation of that producer),
• a monotonic sequence number per partition (which batch in order).

The broker, in ProducerStateManager, remembers the last sequence numbers it has seen for each (PID, partition). When a batch arrives, it checks:

This is why enabling idempotence forces three other settings, which the config doc spells out: acks=all (a write must be committed to be deduped reliably), retries > 0 (there's no point deduping retries you don't make), and max.in.flight.requests.per.connection <= 5 (the broker only tracks the last five batches per partition, so more in-flight than that could slip a duplicate past the window). Modern Kafka turns idempotence on by default, so you likely already have this.

But look hard at the scope, because it's much smaller than "exactly-once" sounds:

Mechanism 2: transactions

Transactions solve the second bug and the parts of exactly-once that actually span your application: making a set of writes across multiple partitions - plus the consumer offset commit - succeed or fail as one atomic unit, and surviving process restarts without duplicating work.

The identity that makes this possible is the transactional.id: a stable name you assign to a logical producer. Unlike the ephemeral PID, the transactional.id is chosen by you and persists across restarts, and that's the whole point - it lets Kafka recognize "the same logical producer, restarted" and fence the old incarnation. Configuring a transactional.id implies idempotence, so mechanism 1 comes along for free.

The transaction coordinator and the two-phase commit

Transactions are arbitrated by a transaction coordinator - a broker, chosen the same way the group coordinator is: your transactional.id hashes to a partition of the internal __transaction_state topic, and that partition's leader is your coordinator. It persists transaction state to that log (so it survives coordinator failover) and drives a two-phase commit through a small state machine: Empty → Ongoing → PrepareCommit → CompleteCommit (or the abort path through PrepareAbort → CompleteAbort).

Here's the full flow of one transactional read-process-write cycle:

Two things in that flow are the entire reason transactions give you application-level exactly-once:

• The offset commit is part of the transaction. In the naive loop, the output write and the commitSync() were two separate operations that a crash could split. sendOffsetsToTransaction (KIP-447) folds the offset commit into the same transaction as the outputs. Now they're atomic: either the results are produced and the input is marked consumed, or neither happened. The crash-between-write- and-commit bug is structurally impossible.
• The epoch bump fences zombies. When a restarted producer calls initTransactions, the coordinator bumps the epoch for its transactional.id. Any older instance still alive with the same id - a "zombie" that hung on a GC pause or a network partition - now carries a stale epoch, and its next transactional write is rejected with ProducerFencedException. Exactly one incarnation of a transactional.id can make progress at a time. That's what makes restarts safe.

The commit is not a rewrite - it's a marker

One detail worth internalizing, because it explains the third mechanism and half the debugging: committing a transaction does not go back and modify the records you wrote. The records were appended to their partitions the whole time, interleaved with everyone else's. What the commit does is write a tiny control record - a COMMIT (or ABORT) marker - into each partition. Aborted records are not deleted; they stay in the log, marked, and it becomes the reader's job to skip them. Which is exactly why there has to be a third mechanism on the read side.

Mechanism 3: read-committed isolation and the Last Stable Offset

The producer side can be flawlessly transactional and a consumer will still read duplicates and aborted garbage - unless it opts into isolation. This is the mechanism people forget exists, because it lives on the consumer and it is off by default.

A consumer has an isolation.level, and the default is read_uncommitted: it returns every record in the log, including ones from transactions that were aborted or are still open. All your careful transactional producing is invisible to it. To see the guarantee, a consumer must set isolation.level=read_committed, and then two things change:

• It filters out aborted records. Each fetch response includes the list of aborted transactions for that range, and the client drops their records before handing anything to your code. (The control markers themselves are never returned either.)
• It only reads up to the Last Stable Offset (LSO) - not the high watermark.

The LSO is the load-bearing concept here, and it's simple: the LSO is the offset of the first still-open transaction. A read_committed consumer will not read past it, because anything at or beyond it might belong to a transaction that hasn't committed or aborted yet - and the consumer refuses to guess.

Putting the three together

Now the naive loop from the top, rebuilt so all three mechanisms are engaged:

You rarely wire this by hand. Kafka Streams packages exactly this behind one setting - processing.guarantee=exactly_once_v2 - which coordinates the consumer offsets, the state-store changelog writes, and the output writes inside a single transaction per commit. The "v2" (KIP-447) matters: the older v1 needed one producer per input partition, which exploded resource use; v2 uses a single producer per instance by sending consumer-group metadata with the offsets, which is the same KIP-447 mechanism you just saw. If you use Streams, you get all three mechanisms correctly composed for free - but knowing what's underneath is what lets you debug it.

Where it bites: the failures each mechanism throws

Now the payoff - the production failures, sorted by which mechanism is involved.

Hung transactions freeze read_committed consumers

Symptom. A read_committed consumer group's lag climbs and climbs, even though the topic's log-end offset is advancing and read_uncommitted consumers on the same partition are perfectly happy.

The tell. The gap between the partition's LSO and its log-end offset is growing. Some transaction started and never committed or aborted - a producer crashed after AddPartitionsToTxn without cleanup, or a delayed network write left a transaction dangling. The LSO is pinned at that open transaction's start, and every read_committed consumer is stuck behind it by design.

The mechanism. #3 (read-committed / LSO) is doing exactly what it should - it cannot advance past an undecided transaction without risking reading uncommitted data. The bug is upstream, on the producer side.

The fix. Keep transaction.timeout.ms sane (default 60s) so the coordinator proactively aborts abandoned transactions instead of leaving them open forever; make sure producers call abortTransaction() in their failure paths; and on modern brokers, KIP-890 (Transactions Server-Side Defense) hardens the coordinator against exactly these hanging transactions with better epoch verification. The instinct: a stuck read_committed consumer is almost never a consumer problem - it's a producer that left a transaction open, and the LSO is the messenger.

ProducerFencedException: your own zombie, or two producers sharing an id

Symptom. A transactional producer dies with ProducerFencedException on commitTransaction or send.

The tell. Something bumped the epoch for this transactional.id after this producer started - meaning another producer with the same id called initTransactions. Two causes: a genuine zombie (this instance hung, a replacement started, and now the old one woke up and is correctly fenced), or a misconfiguration where two live producers were accidentally given the same transactional.id.

The mechanism. #2 (transactions) working as designed - it guarantees exactly one active producer per transactional.id, and it enforces that by fencing.

The fix. Get the transactional.id scheme right: it must be stable across restarts (so a restarted instance reclaims its identity and fences its own zombie) but unique across concurrent instances (so two live producers never collide). For a partitioned consume-produce app, derive it from something stable per logical task - not a random UUID (which defeats zombie fencing) and not a shared constant (which causes the collision). If you see a fencing avalanche, you've usually got a shared or unstable id. The instinct: ProducerFencedException is the system keeping its one-producer promise - the question is only whether the fenced producer was a real zombie or a naming mistake.

"We have exactly-once but we still see duplicates"

Symptom. Everything is transactional on the producer, yet a downstream consumer sees duplicates or aborted data.

The tell. Two usual causes. Either the consumer is on the default isolation.level=read_uncommitted (mechanism #3 never engaged - the most common miss), or the "downstream" is an external system.

The mechanism boundary. This is the most important limit to internalize: Kafka's exactly-once is Kafka-to-Kafka. The transaction spans Kafka partitions and the Kafka offsets topic - nothing else. If your processor also writes to Postgres or calls a payment API, that side effect is not in the transaction and can happen twice on a retry or a fenced commit.

The fix. For the consumer case, set read_committed. For the external-system case, you need an idempotent sink or the transactional-outbox pattern - make the external write safe to repeat, because Kafka can't make it atomic with the log. The instinct: before trusting exactly-once, ask two questions - is the consumer read_committed, and does the pipeline touch anything that isn't Kafka?

OutOfOrderSequenceException

Symptom. An idempotent producer throws OutOfOrderSequenceException.

The tell. The broker saw a gap in the per-partition sequence numbers - a batch it expected never arrived, often because a batch was dropped after exhausting retries (a delivery.timeout.ms expiry) while later batches went through.

The mechanism. #1 (idempotence) refusing to write records out of order, which would break the very guarantee it exists to provide.

The fix. Usually it's a symptom of an underlying delivery failure - investigate that (broker availability, timeouts), not the exception itself. It's rare with default settings and generally means something upstream dropped a batch.

The throughline

"Exactly-once" is the most oversold two words in streaming, not because Kafka doesn't deliver it but because the phrase hides that it's three mechanisms, not one - and they don't come as a set.

The idempotent producer removes duplicates from retries, within one session and partition, and it's on by default - which is why people over-trust it. Transactions remove duplicates from reprocessing, by making your output writes and the input offset commit one atomic unit and fencing zombie producers across restarts - and they're opt-in, gated behind a transactional.id and a very deliberate API. Read-committed isolation makes the first two visible, by holding consumers to the Last Stable Offset and filtering aborted records - and it's off by default, so half the guarantee ships disabled.

Get all three engaged and pointed at the same Kafka-to-Kafka pipeline and you have genuine end-to-end exactly-once. Miss any one - a read_uncommitted consumer, a shared transactional.id, an external side effect outside the transaction - and you have something weaker that looks exactly-once until the day it doesn't. So when someone says "we're exactly-once," the useful questions aren't about whether the feature is on. They're: which of the three are you using, is the read side read_committed, and does anything in this pipeline live outside Kafka? Answer those and you know precisely what you're guaranteed - and precisely where the next duplicate will come from.

This is the third of a trilogy on Kafka's guarantees, each with the same shape - a guarantee you rely on is defined against a live, changing set, and knowing which set is the whole game. The other two: consumer group rebalancing (who owns which partition) and acks=all and the ISR (what counts as durable). For the foundations underneath all three - partitions, offsets, delivery semantics - that's the ingestion and transport curriculum.