Debugging Kafka Consumer Group Rebalancing: Nine Failures and the Clock Behind Each (Part 2 of 2)

Part 1 built the machine: a consumer group is a set of partition leases, and every member is watched by two clocks -

• the session / heartbeat clock (session.timeout.ms, default 45s), which asks is the process alive? and fires on crashes, kills, GC pauses, and network partitions;
• the poll clock (max.poll.interval.ms, default 5 min), which asks is it keeping up? and fires when a healthy process isn't calling poll() fast enough.

A rebalance is triggered by exactly four things: one of those clocks firing, an intentional join/leave, or a partition-count change. That's the whole diagnostic tree. This part is nine real failures walked through it. For each: the symptom, the tell, the clock, and the fix - and the fix is usually not the config a quick search hands you.

The single most useful habit before any of this: turn on the log line that names the cause. Modern Kafka clients log, on every rebalance, the member that triggered it and why. Grep your consumer logs for Revoke previously assigned partitions, Request joining group due to, and Attempt to heartbeat failed - those three phrases carry the reason, and the reason names the clock.

1. The five-minute rebalance loop

Symptom. The group rebalances on a near-perfect cadence - roughly every five minutes. Lag climbs, resets, climbs, resets: a sawtooth. Throughput averages out to a fraction of capacity because the group spends much of its life in PreparingRebalance instead of consuming.

The tell. Five minutes is max.poll.interval.ms's default. A metronome at exactly the poll-clock default is the poll clock firing. Confirm it in the logs:

INFO  o.a.k.c.c.internals.ConsumerCoordinator - [Consumer clientId=...,
  groupId=orders] Member consumer-3 sending LeaveGroup request to coordinator
  due to consumer poll timeout has expired. This means the time between
  subsequent calls to poll() was longer than the configured
  max.poll.interval.ms, which typically implies that the poll loop is spending
  too much time processing messages.

The clock. Poll. The heartbeat thread was healthy the whole time - the process never died. One member took too long between polls, removed itself, triggering a rebalance; after rejoining it fell behind again and repeated.

The fix - and the trap. The popular answer is "increase max.poll.interval.ms." That treats the symptom and often deepens the problem: a consumer that genuinely can't keep up now holds its partitions longer while stalled. The real question is why one poll's worth of records takes minutes to process. Usually it's too many records per poll times too-slow per-record work. The right first move is almost always to shrink the batch:

# Process fewer records per poll so the loop returns in time.
max.poll.records=100          # down from 500
# Only if processing is legitimately long and bounded, raise the ceiling too.
max.poll.interval.ms=600000   # 10 min

Then attack the per-record cost: batch the downstream writes, parallelize processing off the poll thread, or use pause()/resume() to keep polling (which resets the clock) while a slow batch drains. The instinct to build: a five-minute loop is never a session-timeout problem, and raising the poll interval is a tourniquet, not a cure.

2. The deploy that spikes lag (and looks like an outage)

Symptom. Every deploy, lag jumps hard across all partitions for tens of seconds, then recovers. Nothing crashed; the graphs just flatline group-wide during each rollout and dashboards page on the lag threshold.

The tell. The pause hits partitions that didn't change owners. That's the signature of eager rebalancing's stop-the-world: when any member joins or leaves, every member revokes every partition.

The clock. Neither - this is an intentional membership change. The rebalance is correct; it's just far more expensive than it needs to be.

The fix. Move the group to cooperative rebalancing so only the partitions that actually change hands pause, and the rest keep flowing. Remember from Part 1 this is a two-deploy rollout - both assignors, then cooperative alone:

# Deploy 1 (already the modern default): supports cooperative, still negotiates eager.
partition.assignment.strategy=org.apache.kafka.clients.consumer.RangeAssignor,\
  org.apache.kafka.clients.consumer.CooperativeStickyAssignor
# Deploy 2, after every member is on the build above:
partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor

Or, on Kafka 4.0+, adopt the new protocol (group.protocol=consumer), where incremental reconciliation is the default and stop-the-world deploys simply don't happen. Either way the lesson is that "lag spikes on deploy" is not a failure to fix with timeouts - it's the eager protocol doing exactly what it promises.

3. CommitFailedException: the offset that arrived too late

Symptom. Processing throws, mid-batch, with the most-Googled exception in Kafka:

org.apache.kafka.clients.consumer.CommitFailedException: Offset commit cannot be
completed since the consumer is not part of an active group for auto partition
assignment; it is likely that the consumer was kicked out of the group. The group
has already rebalanced and assigned the partitions to another member.

The tell. The consumer tried to commit an offset for a partition it no longer owns. Between two polls, it was kicked out and the partition was reassigned; its commit carries a stale generation and is fenced (Part 1's generation-as-a-fence, doing its job).

The clock. Almost always the poll clock, and #3 is really #1 wearing a different face: the consumer blew max.poll.interval.ms, got removed, and only found out when it came back to commit. The CommitFailedException is the downstream symptom of a poll-clock eviction.

The fix. Same root cause, same fix as #1 - make the poll deadline comfortably: smaller max.poll.records, faster processing, offload heavy work. Two things not to do: don't wrap the commit in a blind retry loop (you're not in the group anymore; retrying commits the same stale offset), and don't switch to session.timeout.ms - the session clock never fired. The instinct: a CommitFailedException is a receipt for a poll-clock eviction that already happened; fix the eviction, not the commit.

4. The rebalance storm during a rolling restart

Symptom. A rolling deploy of a 20-pod group produces not one rebalance but a cascade - dozens, back to back, for minutes after the deploy "finished." The group thrashes long after every pod is healthy.

The tell. Count them: N pods restarting dynamically causes up to 2N rebalances (one when each leaves, one when each rejoins), and if they restart while the group is still mid-rebalance, each one restarts the clock. It compounds.

The clock. Session, indirectly - each dynamic member's identity dies and is reborn. But the deeper issue is that the identity is ephemeral, so a healthy restart looks like a death-plus-birth.

The fix. Static membership. Give each pod a stable group.instance.id (commonly derived from the StatefulSet ordinal or pod name) and a session.timeout.ms comfortably larger than a restart takes. Now a restart within the session window is recognized as the same member, the same partitions are handed back, and no rebalance happens at all. Pair it with your orchestrator's graceful-shutdown hook so the consumer calls close() cleanly. This turns a 2N storm into zero.

5. Scaling up made lag worse

Symptom. Lag was climbing, so someone added consumers to drain it faster - and lag briefly got worse right after, sometimes with no lasting improvement.

The tell. Two sub-cases, and you have to check the partition count to tell them apart:

• Eager cost: adding a member triggered a stop-the-world rebalance, so the whole group paused right when it could least afford to. The added capacity eventually helps, but the rebalance itself spiked lag first.
• The ceiling: if you scaled past the partition count, the extra consumers sit idle. A group can never have more actively-consuming members than the subscribed topic has partitions - member #9 on an 8-partition topic owns nothing and just adds a member to rebalance around.

The clock. Neither - structural. This is a capacity-planning fact, not a timeout.

The fix. Right-size to the partition count: parallelism is capped there, so if you're maxed on partitions, more consumers won't help - you need more partitions (which is itself a rebalance trigger, so do it deliberately). Use cooperative or the new protocol so the scale-up doesn't stop-the-world. And scale in steps, not all at once, so you pay one rebalance instead of one per added pod. The instinct: a consumer group's parallelism ceiling is its partition count, and scaling is a rebalance, so more consumers is not a free lag lever.

6. The GC pause that flaps the group

Symptom. A group rebalances at irregular, unpredictable intervals - no five- minute metronome, no deploy correlation. It just flaps, usually on the same one or two members, often under load.

The tell. It correlates with stop-the-world GC pauses or host CPU saturation. A long GC pause freezes the entire JVM - including the heartbeat thread - so the coordinator stops hearing heartbeats and, if the pause exceeds session.timeout.ms, declares the member dead.

The clock. Session. This is the pure case: the process is "alive" in the sense that it will resume, but for the duration of the pause it is indistinguishable from a crash, because the thing that proves liveness (the heartbeat) is frozen too.

The fix. Two fronts. First, make the pauses smaller: tune the collector (G1 or ZGC with a low pause target), right-size the heap, and stop the allocation storm that's causing full GCs - often the same oversized max.poll.records from #1, since a giant batch is a giant allocation. Second, make the clock more forgiving of transient freezes: a modestly larger session.timeout.ms tolerates a brief pause without evicting. Do not just crank the session timeout to hide chronic GC - you'll delay detection of real crashes. Fix the pauses; widen the window only for the residual jitter.

7. The zombie static member

Symptom. After adopting static membership (from #4), a new failure appears: either a member can't join with FencedInstanceIdException, or - worse - two processes appear to fight over the same partitions.

The tell. Two live processes are configured with the same group.instance.id. Static membership guarantees one instance per id; the coordinator fences the newcomer (or the older one) to preserve that invariant. Common causes: a blue/green deploy where old and new pods overlap, a scale event that reused ordinals, or copy-pasted config.

The clock. Neither - it's an identity collision, the price of the stability static membership buys. The same mechanism that skips rebalances on restart must reject a duplicate identity, or it couldn't make its one-instance promise.

The fix. Guarantee group.instance.id uniqueness and clean handoff: derive it from a stable, unique source (StatefulSet ordinal, not a random UUID that changes per restart - that would defeat static membership entirely), and ensure old pods fully terminate before new ones with the same id start (respect terminationGracePeriod, call close()). The instinct: static membership trades "restarts are free" for "identity must be unique and handed off cleanly" - if you can't guarantee the second, you don't get the first.

8. The coordinator moved out from under you

Symptom. A brief, group-wide rebalance ripples through consumer groups during a broker restart or partition-leadership change - even though no consumer deployed or crashed.

The tell. Consumer logs show coordinator churn, not member churn:

INFO  o.a.k.c.c.internals.AbstractCoordinator - Group coordinator broker-2:9092
  (id: 2 rack: null) is unavailable or invalid due to cause: coordinator
  unavailable. Rediscovery will be attempted.

The clock. Neither of the member clocks. The group's coordinator is the broker leading its __consumer_offsets partition (Part 1); when that broker restarts or the leadership moves, clients must re-run FindCoordinator and re-establish the group, which surfaces as a short rebalance.

The fix. Mostly, don't panic - this is expected and brief, and clients recover on their own. Make it smoother: ensure __consumer_offsets is healthily replicated so leadership fails over cleanly, do broker restarts one at a time with full recovery between them, and keep clients current (newer clients handle coordinator moves more gracefully). The instinct worth building: not every rebalance originates in a consumer - a rebalance with no membership change and coordinator log churn is the broker side moving, and the fix lives in broker ops, not consumer config.

9. The sticky assignor that wasn't sticky (state thrash)

Symptom. A stateful consumer group (Kafka Streams, or a consumer with local RocksDB/caches) rebuilds enormous local state after every rebalance, so each rebalance costs minutes of restore even when membership barely changed. Rebalances that should be cheap are catastrophically expensive.

The tell. You're on an assignor that reshuffles partitions unnecessarily - RangeAssignor or RoundRobinAssignor - so a member that kept doing the same work still gets different partitions after a rebalance and has to rebuild the state tied to them.

The clock. Neither directly, but this is what turns any clock firing into a disaster: the rebalance itself might be legitimate and brief, but a non-sticky assignment multiplies its cost by forcing state rebuilds.

The fix. Use a sticky assignor - CooperativeStickyAssignor for plain consumers, and for Kafka Streams the high-availability sticky task assignor plus standby replicas (num.standby.replicas), which keep a warm copy of state on another instance so a moved task resumes almost instantly instead of restoring from the changelog. Sticky assignment is the difference between a rebalance that costs milliseconds and one that costs the time to replay a state store. The instinct: for stateful consumers, minimizing partition movement matters as much as minimizing the pause - a cheap rebalance with an expensive reassignment is still an expensive rebalance.

The one table to keep

Nine failures, three clocks (counting "neither"), and a consistent pattern: the symptom (lag spikes, endless rebalancing) is nearly useless for diagnosis because several unrelated causes produce it. The clock is what discriminates.

Closing the loop

You now have both halves. Part 1 is the machine - leases, two clocks, the coordinator handshake, eager vs cooperative, static membership, and the KIP-848 shift to server-side, epoch-driven reconciliation. This part is the machine under fire: nine failures, each diagnosed by which clock fired and fixed by the knob that clock owns rather than the one the internet reaches for first.

The whole point of the two-clock framing is that it makes rebalancing boring - in the best way. A boring incident is one where you already know the shape of the answer before you open the dashboard. Next time a group starts flapping, don't start with a config. Start with the question: which clock fired? Everything follows from that.

If you want the streaming foundations underneath all of this - partitions, offsets, delivery semantics, and how ingestion pipelines are actually built and operated - that's the ingestion and transport curriculum, which teaches the model these two posts assume.