A Kafka Transaction Is a State Machine Living in a Log You Never Read

Here is the sentence almost every explanation of Kafka transactions gives you, and it is true: a transaction lets a producer write to multiple partitions atomically, all-or-nothing. The Kafka producer's own javadoc says exactly that - "The transactional producer allows an application to send messages to multiple partitions (and topics!) atomically."

The problem is that the sentence quietly implies a mental model that is wrong, and the wrong model is the source of nearly every transactions bug. The implied model is: the broker holds your records in some pending area, and at commit time it makes them real. A staging buffer. A lock. Something that keeps uncommitted data out of the log until you say go.

None of that happens. Your transactional records are appended to their partitions the moment you send them - written to the log, assigned offsets, replicated to followers, interleaved on disk with records from every other producer, transactional or not. They are physically there before you commit. If a consumer is configured to read them, it can. The transaction is not a wall around your writes.

So what is it? It is a completely separate machine bolted onto the side of the log:

• a transaction coordinator - a broker running a two-phase-commit state machine,
• whose state is itself persisted to an internal replicated log (__transaction_state),
• which, at the end, stamps a single tiny control marker into each partition you touched,
• and a read-side rule that teaches consumers to honor those markers and skip what was aborted.

Commit does not go back and modify your records. It writes one marker per partition and moves on. That single design decision - the data is already in the log; only its verdict is pending - explains the whole protocol, every failure mode, and the one concept (the Last Stable Offset) that ties the write side to the read side. This post builds that machine from the Apache Kafka source up.

The identity: transactional.id and why it outlives the process

Everything starts with one config, and it is worth being precise about what it does, because it is a different kind of identity from the one idempotence uses.

The idempotent producer gets a Producer ID (PID), an integer the broker hands out. It is ephemeral: restart the producer and you get a brand new PID that remembers nothing. That is fine for deduping retries inside one session, and useless for anything that has to survive a crash.

Transactions need identity that persists across process restarts, because the entire point is to recover cleanly when a producer dies mid-transaction. That identity is the transactional.id, a name you choose. The producer javadoc is unusually explicit about both its purpose and how to pick it:

Read that twice, because it contains the entire operational contract in two clauses that pull in opposite directions. Stable across restarts (so a restarted instance reclaims its identity and can fence its own zombie) but unique across concurrent instances (so two live producers never share one). Get that scheme wrong in either direction and you get one of the two failure modes at the end of this post. Setting a transactional.id also flips idempotence on automatically along with the configs it depends on (acks=all, retries, bounded in-flight), so the transactional producer is always also an idempotent one.

The coordinator, and the log it runs on

A transaction is arbitrated by a transaction coordinator. It is not a separate service - it is a broker, wearing a second hat, chosen the same way a consumer group's coordinator is chosen: your transactional.id is hashed to a partition of an internal topic called __transaction_state, and the leader of that partition is your coordinator.

This detail is not trivia. It is what makes the coordinator itself fault-tolerant. The coordinator does not keep transaction state only in memory - it writes every state change to that __transaction_state partition, which is a normal Kafka log, replicated with the same ISR and high-watermark machinery as any topic. So when a coordinator broker dies, another broker becomes leader of that partition, replays the log, and reconstructs every in-flight transaction's state exactly where it left off. The two-phase commit is crash-safe because its state machine is durable for the same reason your data is durable: it is on a replicated log.

That is the recurring shape of this whole series - a guarantee you rely on turns out to be defined against another replicated log underneath it. Here the transaction's fate is itself just committed log records.

The protocol, RPC by RPC

Now walk one read-process-write transaction the way the client and coordinator actually walk it. This is more granular than the "one transaction, end to end" sketch in the exactly-once post, because the individual RPCs are where the state machine and the failures live.

Two things in that flow are worth stopping on.

First, the coordinator only ever handles control, never data. Your records never pass through it. It maintains a set - "the partitions this transaction has touched" - and its entire job at commit time is to write one marker into each member of that set. That is why a transaction with a thousand records across ten partitions commits in roughly the time it takes to write ten tiny markers: the expensive part (the records) already happened.

Second, PrepareCommit is the decision, and it is durable before any marker is written. This is the "two phases" of two-phase commit. Phase one: persist the intent-to-commit and the partition set to __transaction_state. Phase two: propagate markers to the partitions. If the coordinator dies between the two, the replacement reads PrepareCommit from the log, sees the transaction was decided but not finished, and re-drives phase two - re-sending the markers. The outcome can never flip from committed to aborted after PrepareCommit is durable. The decision and its execution are deliberately separated so a crash in the middle is always recoverable in one direction.

The state machine, exactly as the coordinator encodes it

The coordinator tracks each transaction through a small set of states, and the source encodes them as a plain enum with an explicit table of legal transitions. This is the actual TransactionState from the transaction-coordinator module, and it is worth seeing the states and what each one is for:

A few things this makes concrete that prose usually blurs:

• Ongoing is reachable from the two Complete states. A long-lived producer does not return to Empty between transactions; it goes CompleteCommit → Ongoing → PrepareCommit → CompleteCommit around the loop. Empty is really just the freshly-initialized state.
• The commit and abort paths are structurally identical, two phases each, differing only in which marker gets written. Abort is not a rollback that undoes writes. It is a commit of the opposite verdict - the same machinery, writing ABORT markers instead of COMMIT markers, leaving the records exactly where they are.
• PrepareEpochFence is the zombie-killer as a first-class state. When a newer producer with the same transactional.id initializes, and an older transaction is still Ongoing, the coordinator does not just ignore the old one - it moves that transaction through PrepareEpochFence into PrepareAbort, force-aborting it, and any further write from the old producer is rejected. Fencing is not a side check; it is a path through the state machine.
• There is also a Dead state, entered when a transactional.id has expired and is being removed from the cache entirely (more on expiration below).

The commit is a marker, not a rewrite

This is the load-bearing idea, so it gets its own section. When the coordinator commits, it writes an EndTransactionMarker - a control record - into each partition. The source describes it plainly:

Watch the whole thing run, and watch what doesn't move. Toggle between commit and abort: the same four steps play either way, the data records stay in the same cells, and only the marker's verdict - and whether the reader keeps or drops the records below it - changes:

Two consequences fall out of "the commit is a marker."

Aborted records are not deleted. When a transaction aborts, the records it already wrote stay in the log, at their assigned offsets, forever (until normal retention removes them). The ABORT marker does not reach back and erase them. It just says "everything from this producer, this transaction, up to here: void." Physically the log still contains them; it is the reader's job to skip them. Which is the entire reason a read-side mechanism has to exist at all.

The marker carries a coordinator epoch. That coordinatorEpoch embedded in the marker value is a fencing token for the coordinator itself. If an old coordinator (deposed during a failover) tries to write a stale marker, the partition leader can reject it by epoch. The same fencing idea that protects against zombie producers also protects against zombie coordinators, one layer down.

The read side: the Last Stable Offset

The write side can be flawless and a naive consumer will still read aborted garbage, because the aborted records are sitting right there in the log. Making the guarantee visible is a separate mechanism that lives on the consumer, and its load-bearing concept is the Last Stable Offset (LSO). Here is the definition straight from UnifiedLog in the source, and it is worth reading literally:

Unpack "decided." A non-transactional record is decided the instant it is written - there is no verdict pending. A transactional record is undecided until its marker lands. The LSO is the boundary: the first offset that is still undecided. Its mirror image in the code is the first unstable offset, and the asymmetry between them is telling - "Unlike the last stable offset, which is always defined, the first unstable offset only exists if there are transactions in progress." When nothing is open, there is no unstable region and the LSO simply equals the high watermark.

Watch it move. This is the read-side twin of the high watermark from the acks post - same record strip, same two boundary markers, but where the HW asks "how many replicas have this?", the LSO asks "has this been decided yet?":

A read_committed consumer reads only up to the LSO. That is the whole guarantee on the read side:

How does the consumer know which records below the LSO to drop? It does not compute that itself. Each segment carries an aborted-transaction index on the broker, and the source spells out its job:

So on a read_committed fetch the broker consults this index, attaches the list of aborted transactions overlapping the fetch range, and the client subtracts their records before handing anything to your code. The index also notes that "individual transactions may span multiple segments," which is why recovering it can require scanning earlier segments to find where a transaction started - a transaction is not confined to one file on disk.

Timeouts, expiration, and who cleans up

A transaction that starts and never finishes is not just a leak - it pins the LSO and freezes every read_committed consumer behind it. So the coordinator has several timers, each a real config with a real default, worth knowing by name:

• transaction.timeout.ms (producer side, default 60s) - the max time the producer expects a single transaction to take. The producer proposes it; the broker caps it.
• transaction.max.timeout.ms (broker side, default 15 minutes) - the ceiling on the above. A producer cannot ask for a longer-lived transaction than the cluster allows.
• transaction.abort.timed.out.transaction.cleanup.interval.ms (default 10s) - how often the coordinator sweeps for transactions that blew their timeout and proactively aborts them. This is the mechanism that unsticks a hung transaction without human intervention: the coordinator itself drives it to PrepareAbort and writes ABORT markers, and the LSO is freed.
• transactional.id.expiration.ms (default 7 days) - how long the coordinator keeps an idle transactional.id's state around with no updates before evicting it (the transition into Dead). Come back after this with the same id and you are effectively a new producer.

The instinct to carry: a stuck read_committed consumer is almost never a consumer problem. It is a producer that left a transaction open, and the LSO is just the messenger. The fix is upstream - sane timeouts so the coordinator aborts abandoned transactions, and producers that call abortTransaction() on their failure paths.

What KIP-890 changed: Transaction V2

Everything above is the classic protocol (KIP-98, with the offset-in-transaction piece from KIP-447). But there was a long-standing class of bug in it - hanging transactions - and fixing it changed the protocol enough to be worth its own version. This is KIP-890, "Transactions Server-Side Defense," and the source is now full of isTransactionV2Enabled branches gated on a transaction.version feature flag.

The classic bug: the coordinator and the partition leader could disagree about whether a given write belonged to the current transaction. A delayed network write from a previous transaction could arrive at a partition after the coordinator had moved on, and get appended as if it were part of the log's normal flow - a record stuck in limbo that no marker would ever cover, pinning the LSO forever. The classic protocol had no way for the partition to verify "is this write actually part of an open transaction the coordinator knows about?"

The mechanism V2 leans on is a per-transaction epoch bump. In V1 the producer epoch changed only on initTransactions (per session). In V2, "the producer epoch is bumped on every transaction," which means every transaction has a distinct epoch, and any record carrying a previous transaction's epoch is now stale and gets fenced by the existing log-layer epoch check. The gap that let a late write slip into the wrong transaction closes, because the write's epoch no longer matches. V2 also lets the client skip the explicit AddOffsetsToTxn round-trip and implicitly add partitions, trimming RPCs. It is "automatically enabled on the server since Apache Kafka 4.0" - so on a modern cluster with modern clients, you are already on it.

This is also the piece that made Kafka Streams' exactly_once_v2 cheap. The older EOS v1 needed one producer per input partition (resource use exploded with partition count); the KIP-447 + KIP-890 work lets a single producer per instance carry consumer-group metadata with the offsets and bump epochs per transaction, which is why v2 scales where v1 did not.

The frontier: KIP-939 and external two-phase commit

One more, because it reframes what a transaction can span. Everything so far is Kafka-to-Kafka: the transaction covers Kafka partitions and the Kafka offsets topic, nothing else. Write to Postgres inside your processor and that write is not in the transaction - it can happen twice on a retry.

KIP-939 cracks that open by letting Kafka be a participant in someone else's two-phase commit rather than always being the coordinator. It adds a producer config, transaction.two.phase.commit.enable (broker default false), described in the source as "Allow participation in Two-Phase Commit (2PC) transactions with an external transaction coordinator." When set, Kafka cedes the fate decision: it will "no longer proactively abort transactions based on timeout, and will also allow resuming previous transactions," because now an outside coordinator - a database's XA manager, or Flink's job manager acting as coordinator (FLIP-319) - owns the commit/abort verdict across both systems. This is the path to a genuine atomic write across Kafka and a database, and it is why the coordinator's timeout-abort behavior had to become optional: an external coordinator might legitimately hold a transaction open across a checkpoint far longer than 15 minutes.

Where it bites: reading the machine when it misbehaves

The payoff of knowing the machine is that its failures stop being mysterious. Three, sorted by which part of the machine is talking.

A hung transaction, and the LSO gap that names it

Symptom. A read_committed consumer group's lag climbs while the log-end offset keeps advancing and read_uncommitted consumers on the same partition are fine.

Read the machine. The gap between the partition's LSO and its log-end offset is growing. Some transaction reached Ongoing and never reached a Complete state - a producer died after AddPartitionsToTxn without aborting, or (on V1) a late write hung. The LSO is pinned at that transaction's first offset, and mechanism #3 is doing exactly what it must: refusing to read past an undecided offset. Use the Admin API - DescribeTransactions / ListTransactions - to find the open transaction and its state directly, rather than guessing.

Fix. Sane transaction.timeout.ms so the coordinator's sweep aborts it; producers that abortTransaction() in failure paths; and on modern clusters, Transaction V2 (KIP-890), which closes the specific gap that produced un-abortable hanging transactions in the first place.

ProducerFencedException: the fence path fired

Symptom. A transactional producer dies with ProducerFencedException on send or commitTransaction.

Read the machine. Something bumped the epoch for this transactional.id after this producer started - i.e. another producer with the same id called initTransactions and drove this one through PrepareEpochFence. Two causes, and telling them apart is the whole diagnosis: a genuine zombie (this instance hung, a replacement took over, the old one woke up and was correctly fenced), or a misconfiguration where two live producers were handed the same transactional.id.

Fix. It is the transactional.id scheme, always. Stable across restarts and unique across concurrent instances - derive it from a stable per-task shard identity, never a random UUID (which defeats zombie fencing by giving the restarted instance a new id) and never a shared constant (which makes two live producers fence each other in a loop). A fencing avalanche is a naming mistake until proven otherwise.

Here is the fence firing, step by step - the PrepareEpochFence path from the state machine as it actually plays out when a second incarnation claims the same transactional.id:

"We're transactional but still see duplicates"

Symptom. The producer side is fully transactional, yet something downstream sees duplicates or aborted data.

Read the machine. Two usual causes. The consumer is on the default isolation.level=read_uncommitted so the read-side mechanism never engaged (by far the most common), or the "downstream" is an external system that was never inside the transaction to begin with - the KIP-939 case, before KIP-939. Kafka's transaction spans Kafka partitions and the Kafka offsets topic and nothing else; a Postgres write or a payment call in your processor is outside it and can repeat.

Fix. Set read_committed on the consumer. For the external system, make the write idempotent or use the transactional-outbox pattern - or, on a modern stack, look at whether KIP-939 2PC actually fits.

The throughline

The reason Kafka transactions feel slippery is that the name points at the wrong place. The word "transaction" makes you look at your records, waiting to be committed. But your records were never the pending thing - they were in the log the whole time. The pending thing is a verdict, tracked by a state machine on a separate replicated log, executed as one marker per partition, and honored (or not) by a reader you configure independently.

Hold that and the pieces stop being a list to memorize and become one machine. The transactional.id is durable identity so the machine can recognize a restarted producer and fence its zombie. The coordinator is a broker running two-phase commit whose state survives failover because it is itself on a log. PrepareCommit is the durable decision that makes the outcome irreversible before any marker is written. The markers are the execution, carrying a coordinator epoch so a deposed coordinator cannot forge them. The LSO is the read-side shadow of PrepareCommit - the boundary of what has been decided. And KIP-890's per-transaction epoch bump exists to close the one seam where a stray write could sneak into a verdict it did not belong to.

So when someone says "we use Kafka transactions," the useful questions are not about whether the feature is on. They are: is the read side read_committed, is the transactional.id both stable and unique, are you on Transaction V2, and does anything in this pipeline live outside Kafka? Answer those and you know precisely what the machine guarantees you - and precisely where the next stuck LSO will come from.

This deep-dive sits under the exactly-once trilogy, which frames transactions as one of three mechanisms alongside the idempotent producer and read-committed isolation. For the durability model the coordinator's own state log is built on - the ISR and the high watermark - see acks=all and the ISR. For the primitives underneath all of it - partitions, offsets, delivery semantics - that is the ingestion and transport curriculum.