Queues for Kafka (KIP-932): The Feature That Isn't a Consumer Group

Here is a rule every Kafka engineer knows well enough to recite in their sleep:

A partition is assigned to at most one member of a consumer group at a time.

Everything follows from it. Partition count is a parallelism ceiling - 10 partitions means at most 10 useful consumers, and the eleventh sits idle. Kafka delivers each partition to one group member in offset order. In the conventional sequential processing model, a slow record therefore blocks the work behind it. Applications can process records concurrently, but then they must track completed gaps themselves and only commit a safe contiguous offset. Adding consumers past the partition count does nothing. Scaling a consumer group beyond that ceiling means adding partitions, which you can't easily undo.

We've all built around that rule for a decade. It reads like a law of physics.

KIP-932 repeals it. In a share group, a hundred consumers can share a ten-partition topic and all hundred get work. You acknowledge individual records, not offsets. A failed record is redelivered without rewinding anything. A record that fails too many times gets archived instead of poisoning the partition forever.

"Queue" needs a boundary here. KIP-932 adds cooperative, per-record delivery semantics to ordinary Kafka topics; it does not create a separate queue resource. Acknowledgement does not delete a record from the log, topic retention still applies, and each share group maintains its own delivery state independently.

And the thing worth understanding is why this could never have been a new assignor. Kafka didn't loosen the partition rule. It went after the thing underneath the rule - and the thing underneath was never the assignment strategy. It was the offset.

The offset is a watermark, and a watermark can't be a queue

A consumer group's entire memory of its progress is one number per partition. A long. __consumer_offsets stores it, the group coordinator manages it, and it means exactly one thing:

That's a watermark. It's a single dividing line, and it's the reason for every consumer group limitation you've ever hit. Watch it fail at queueing:

You fetch records 4, 5, 6, and 7. Records 5, 6, and 7 process fine in 2ms. Record 4 calls a payment API that's timing out and takes 30 seconds. What do you commit?

• Commit 8? You just marked record 4 as done. If you crash, it's gone forever. Data loss.
• Commit 4? You failed to record the completed work on records 5, 6, and 7. On restart they're redelivered and reprocessed. Duplicates.
• Wait for record 4? That's head-of-line blocking - the thing you're trying to escape.

There is no right answer, because the question is unanswerable in the data model. You want to express "5, 6, 7 done; 4 still in progress." That's a set. You have a number. No assignor, no rebalance protocol, no client library can fix that, because the information physically cannot be written down.

The move: state goes to the broker

So share groups make one change, and everything else in this post is downstream of it.

Consumer group: progress is a number, and the client owns it. The client decides what to commit and when. The broker just stores whatever the client says.

Share group: progress is a per-record state machine, and the broker owns the authoritative state. The client records acknowledgements - individually in explicit mode or for the previous poll as a batch in implicit mode - then sends them with commitSync(), commitAsync(), or a subsequent poll(). What it commits is no longer one progress offset. It is the outcome of each delivered record.

That inversion is the whole feature. Once the broker tracks each record individually, "which consumer exclusively owns this partition?" stops being the right question. The group coordinator still assigns share-partitions to members, and membership changes can rebalance those assignments. The difference is that a partition may be assigned to multiple members at once. Those consumers acquire records, and the broker can hand out different records from the same partition because it knows which ones are already out.

The state machine

Every in-flight record has a state. The core states are:

And the transition rules, enforced by validateTransition:

if (this == ACKNOWLEDGED || this == ARCHIVED) {
    throw new IllegalStateException("The state transition is invalid from the current state: " + this);
}
if (this == AVAILABLE && newState != ACQUIRED) {
    throw new IllegalStateException("The state can only be transitioned to ACQUIRED from AVAILABLE");
}
// Either the transition is from Available -> Acquired or from Acquired -> Available/
// Acknowledged/Archived.

Two rules, and they're both load-bearing:

1. ACKNOWLEDGED and ARCHIVED are terminal for this share group's normal delivery. Fetching will not acquire those records again for the group. Their explicit state can be discarded once the SPSO advances past them.
2. AVAILABLE can only go to ACQUIRED. You cannot acknowledge a record you don't hold. Delivery is the only way in.

So the Kafka 4.2 legal universe is: AVAILABLE → ACQUIRED, then ACQUIRED → {AVAILABLE, ACKNOWLEDGED, ARCHIVED}. A record loops between AVAILABLE and ACQUIRED as many times as it takes, then exits normal delivery. That loop is redelivery, and it is the thing a consumer group offset could never express.

The acquisition lock: delivery is a lease

Here's the mechanism that makes it safe to hand the same partition to many consumers.

When a consumer acquires a record, the record goes ACQUIRED and starts a timer. The broker default is group.share.record.lock.duration.ms=30000. A group can override it with share.record.lock.duration.ms. The broker accepts an absolute range of 1 second to 1 hour; by default, group.share.min/max.record.lock.duration.ms narrows the permitted group override to 15 to 60 seconds. While the lock is held, that record is invisible to every other consumer in the same share group.

If the consumer sends ACCEPT or REJECT before the lock expires, the record hits a terminal state and the lock is released; RELEASE makes it available and RENEW extends the lock. If the consumer says nothing - it crashed, it hung, it GC'd for a minute, the network ate the response - the lock expires. The record goes back to AVAILABLE when its delivery count is still below the limit; once the count has reached the limit, it goes to ARCHIVED instead.

This is a lease, similar to the visibility-timeout model in SQS. It's the difference between "this consumer owns this partition until a rebalance says otherwise" and "this consumer holds this record for the next 30 seconds."

Consider what that buys you. A share consumer still joins through heartbeats, and its failure still causes the group coordinator to update share-partition assignments after the session timeout. What changes is record recovery. The failed member did not exclusively own a partition or its progress watermark. Its acquired records become available as their locks expire, while other members assigned the same share-partition can keep working. Compare that to the machinery a consumer group rebalance needs to restore exclusive partition ownership.

Four ways to say what happened

The consumer API is where the model surfaces. AcknowledgeType, straight from the source comments:

RELEASE and REJECT are the pair that matters, and the distinction is one your code already knows how to make:

• RELEASE = transient failure. The payment API returned 503. Try again, maybe on another consumer, maybe in a moment. Retryable.
• REJECT = permanent failure. The record is malformed JSON. It will fail forever on every consumer. Don't waste another attempt on it.

That's the poison-pill distinction, expressed as an API call. In a consumer group you have to build this yourself, badly - usually a try/catch that produces to a DLQ topic and commits anyway, with all the dual-write hazards that implies.

RENEW deserves its own note because it's the escape hatch for long processing. Rather than raising the lock duration cluster-wide for every record because one workload is slow, a consumer that's still working says "not done, still alive, give me more time." It's a heartbeat scoped to a single record.

Poison pills die on their own

Every record carries a delivery count. Each acquisition increments it. If an attempt fails or its lock expires after the count has reached the broker setting group.share.delivery.count.limit (default 5, bounded 2-10), the broker moves the record to ARCHIVED instead of making it available again.

That gives the broker a bounded escape hatch for poison pills. A record that repeatedly fails gets at most five delivery attempts by default, then leaves circulation. This does not make processing code unnecessary: consumers still need to classify errors, make handlers idempotent, and decide how rejected data will be inspected or recovered. Archiving is scoped to this share group and does not delete the original record from the Kafka topic or prevent another group from consuming it.

The delivery count is deliberately a safety mechanism, not an audit counter. Kafka does not persist every state update with exactly-once semantics, so the count is not guaranteed to be precise in every failure scenario.

In a consumer group, this scenario is a genuine incident: a bad record throws, the consumer doesn't commit, it restarts, fetches the same record, throws again - a crash loop that blocks that partition until a human intervenes and manually seeks past the offset. Every mature Kafka shop has written retry-count scaffolding by hand, usually more than once. Share groups make the redelivery ceiling a broker configuration. In Kafka 4.2 and 4.3, however, ARCHIVED means the record will not be delivered again; it is not automatically copied to a dead-letter topic.

The SPSO: how a queue still has an offset

If records complete out of order, how does the broker ever forget anything? It can't keep per-record state for a topic with a trillion records.

Enter the Share Partition Start Offset (SPSO). It's the offset below which everything is terminal - every record before it is ACKNOWLEDGED or ARCHIVED, so none of them can ever be delivered again to this share group, so none of their states need to be remembered.

The SPSO looks like a consumer group's committed offset, and confusing the two is the mistake to avoid. The difference:

• A committed offset is a decision. The client chose it, and everything after it is presumed undone.
• The SPSO is a consequence. Nobody sets it. It's derived - it advances on its own whenever the records at the boundary reach terminal states.

There are two administrative caveats to "nobody sets it." An administrator can reset the SPSO while the share group is empty, which discards its in-flight state and delivery counts. And the topic's log start offset is a hard lower bound: retention can move the SPSO forward and archive records that were still eligible for delivery. A size-based retention policy can therefore remove unprocessed queue work. Share groups add delivery state; they do not override the topic's retention policy.

There is also an initialization default worth making explicit: the first time a share group subscribes to a topic, its SPSO starts at the latest offset unless the group's share.auto.offset.reset setting or an administrative reset selects another position.

Between the SPSO and the Share Partition End Offset (SPEO) sits the in-flight window: records in mixed states, some acquired, some available, some already acknowledged out of order. That window is the queue's remembered state. The broker setting group.share.partition.max.record.locks (default 2000, despite the historical name) bounds the total SPSO-to-SPEO in-flight window for one share-partition. It does not count only records currently in ACQUIRED.

One unresolved record at the SPSO boundary therefore has a direct cost. Suppose record 4,102 is being retried while the next 1,999 records are acknowledged. The SPSO cannot advance past 4,102, so all 2,000 offsets remain in the in-flight window. Kafka may reacquire an AVAILABLE record already inside that window, but it cannot extend the SPEO to fetch a new higher offset until the boundary gap closes and the SPSO advances.

Where the state actually lives

The durable part of per-record state has to survive a broker restart, so it is persisted in a new internal topic, by a new coordinator. The important exception is ACQUIRED: acquisition locks are transient and that state is not persisted. The durable representation contains the SPSO plus the state batches and delivery-count information needed to reconstruct the share-partition.

The topic is __share_group_state (50 partitions by default, RF 3, min.insync.replicas=2). The component is the share coordinator, and it is not the group coordinator. The group coordinator still handles share group membership; the share coordinator handles share state. Two different coordinators for one feature, which is exactly why "a share group is just a consumer group with a different assignor" is wrong at every level of the stack.

The interesting engineering is in how state is written. The naive design - one record per record - would be catastrophic: acknowledging 500k records/sec would mean 500k state writes per second. Instead, PersisterStateBatch stores ranges:

private final long firstOffset;
private final long lastOffset;
private final short deliveryCount;
private final byte deliveryState;

"Offsets 4,103 through 6,102 are all ACKNOWLEDGED with delivery count 1" is one batch, not 2,000. Contiguous runs therefore stay compact; highly fragmented completion patterns create more ranges. It is run-length encoding, and it is the trick that makes per-record state practical at Kafka's scale. (PersisterStateBatchCombiner merges adjacent ranges as they form.)

The topic uses a snapshot-plus-update pattern that is conceptually similar to KRaft recovery: ShareUpdateValue records are incremental updates, and every share.coordinator.snapshot.update.records.per.snapshot (default 500) updates, a full ShareSnapshotValue record is written to bound replay work. This is not the same storage mechanism as a KRaft snapshot, but it serves the same recovery purpose described in KRaft snapshots: bound how much of an update log must be replayed to reconstruct state.

What you gave up

Now the honest part, and it's the reason share groups aren't simply better.

Share groups do not guarantee processing order across deliveries.

Kafka does preserve one narrow ordering property: records returned in one batch for one share-partition are in increasing offset order. There is no offset-order guarantee between batches. Records from one partition can go to different consumers simultaneously, so record 5 may be processed before record 4. A RELEASE can put record 4 back in the pool to be redelivered after records 5 through 500 are done. The batch-local delivery order is not an application-level processing-order guarantee.

That is the central trade. Partition-consumer coupling was never gratuitous; it was how Kafka bought you ordering. One active reader per partition is what lets a single-threaded consumer process same-key records in order. Share groups allow concurrent processing within a partition, so applications cannot rely on that processing order.

So the decision is clean:

Notice that the share group column is a list of things that were never really log problems. They're task queue problems - work items that happen to arrive via a log. That's the niche: for workloads that only need concurrent work distribution, acknowledgements, and bounded redelivery, share groups may remove the need for a separate queue broker. They do not reproduce every routing, scheduling, priority, or retention feature of systems built specifically as queues.

Ops and debugging

Share groups are production-ready as of Kafka 4.2. The path there was deliberately slow: early access in 4.0, preview in 4.1, and production-ready in 4.2. Kafka 4.1 required an explicit upgrade to share.version=1; follow the upgrade notes for the exact broker and feature-level sequence when moving between releases. If you are starting on 4.2, use at least 4.2.1: it fixes a reproducible potential deadlock in the share-partition path.

The first three values below are broker defaults for dynamic share-group settings. The last two configure the share coordinator itself:

The two failures worth building instincts for:

"Consumers are idle but there's a backlog." Check whether the share-partition has filled its SPSO-to-SPEO in-flight window, whether acquired records are waiting for long locks to expire, whether assignments and heartbeats are healthy, and whether fetch or acknowledgement requests are failing. A non-advancing SPSO identifies an unresolved gap, but does not by itself prove that the lock limit is exhausted.

"Records are processed twice." Expected, and by design: share groups are at-least-once. Share groups can use the group-level share.isolation.level=read_committed setting to avoid delivering aborted transactional records. That is input isolation, not an end-to-end exactly-once processing guarantee. A consumer that performs an external side effect and then dies before its acknowledgement is committed will have the record redelivered when the lock expires. Your handlers must be idempotent. If you need Kafka's transactional consume-process-produce pattern, you need a consumer group and the transactional path; share-group acknowledgements cannot participate in that transaction.

And keep an eye on __share_group_state itself. It's a real topic on real brokers, and its health is your share groups' health. A share coordinator that can't write state is a share group that can't acknowledge anything.

The reframe

Share groups are the least Kafka-shaped thing Kafka has ever shipped, and that's the point.

Every other feature in the system is a variation on one idea: an ordered, immutable log that consumers read at their own pace, tracking a position. Share groups keep the log and throw out the position. The broker stops being a dumb byte pipe and becomes a work-distribution engine with per-item leases and retry counts. The underlying storage is still a Kafka log; the delivery behavior is queue-like.

Kafka spent fifteen years telling people "you don't want a queue, you want a log," and it was right often enough to win. But it was never right always, and the honest read on KIP-932 is that Kafka finally admitted a big chunk of what people were doing with consumer groups - fan out tasks to workers, retry the failures, retire the poison - was queue work being done by a log, badly, with a try/catch and a DLQ topic and a lot of hope.

The log was never the wrong abstraction. It was just never the only one.

Share groups are the newest of the four mechanisms that decide how a Kafka cluster behaves under load. For the model they replace - and the rebalance protocol they route around - see the two-part series on consumer group rebalancing. For what makes a write durable enough to be worth delivering, see acks=all and the ISR. For the related snapshot-plus-update recovery pattern used by __share_group_state, see KRaft. The foundations are in the ingestion and transport curriculum.

Primary sources: KIP-932, the Kafka 4.2 release announcement, the Kafka 4.2.1 deadlock fix, the Kafka 4.2.1 source for SharePartition, GroupConfig, and ShareGroupConfig, plus the accepted KIP-1191 DLQ proposal.