How Kafka Consumer Group Rebalancing Actually Works: The Two Clocks (Part 1 of 2)

A consumer group starts flapping at 3am. Lag climbs, resets, climbs again - a sawtooth crawling up the dashboard. On call, you do what everyone does: reach for max.poll.interval.ms, double it, ship. Sometimes the sawtooth flattens and you go back to bed. Just as often it does nothing, or the next incident is worse - because the setting you changed had no connection to the thing that actually fired.

That gap, between the config people reach for and the mechanism that broke, is what this post closes. Rebalancing gets learned as folklore - a bag of timeouts you tune by superstition - when underneath it is a small, legible machine.

Here is the claim this whole post is built to earn: a rebalance is never a mystery. It is always one of exactly four things -

1. A member's session/heartbeat clock expired (the coordinator thinks the process died).
2. A member's poll clock expired (the process is alive but not calling poll() fast enough, so it removes itself).
3. A member intentionally joined or left (scale up, scale down, deploy).
4. The partition count of a subscribed topic changed.

Two of those four are clocks. Almost every painful, recurring, "why is this group stuck rebalancing every five minutes" incident is a clock firing, and the fix depends entirely on which clock. So we are going to build the machine from the bottom up - the coordinator, the handshake, the two clocks, the assignors, and the new protocol - and by the end you'll be able to look at a rebalance and name its cause before you open a dashboard. The debugging playbook is Part 2; this part is the mental model it stands on.

The primer: a consumer group is a lease, not a connection

Start with what a consumer group is, because the usual one-liner ("it lets multiple consumers share a topic") hides the mechanism that matters.

A topic is split into partitions - the unit of parallelism and ordering. A consumer group is a set of consumer instances that share a group.id and agree to divide those partitions among themselves so that each partition is owned by exactly one member of the group at a time. That exclusivity is the entire point. It is what lets Kafka promise in-order processing per partition and lets you scale out by adding members.

But "owned by exactly one member at a time" is a distributed-systems problem in disguise. Members are on different machines. They crash, GC-pause, get network-partitioned, and deploy in rolling waves. Nobody has a global view. So ownership can't be a static config - it has to be a lease that is granted, renewed, and revoked. Rebalancing is the protocol that re-grants those leases whenever the set of members or partitions changes.

The unit being leasedone topic, four partitions, three members

Topic

ordersP0 · P1 · P2 · P3

Group members (owners)

consumer-Aowns P0, P1consumer-Bowns P2consumer-Cowns P3

Each partition is leased to exactly one member. Add or remove a member and the leases must be re-cut - that re-cutting is a rebalance.

Because a lease has to be renewable, every member is on a heartbeat. And because "the process is up" is not the same as "the process is making progress," Kafka watches two different things about each member. Those are the two clocks.

The two clocks (the one idea to keep)

This is the load-bearing section. Everything after it is detail.

Kafka is trying to answer two different questions about every member, and it uses two independent timers to answer them:

Two questions, two clocksthe framing the rest of the series hangs on

Is this member OK?the coordinator, continuously

which is answered by two independent timers

is it alive?The session / heartbeat clockA background thread sends heartbeats every heartbeat.interval.ms (default 3s). If the coordinator sees no heartbeat within session.timeout.ms (default 45s), it declares the member dead and rebalances. This clock catches crashes, kills, and network partitions - things that stop the whole process.

is it keeping up?The poll clockThe consumer must call poll() at least every max.poll.interval.ms (default 5 min). If your processing loop takes longer than that between polls, the member proactively LEAVES the group. This clock catches a live process that is stuck or too slow - heartbeats are fine, but no records are moving.

The reason this split exists is subtle and worth sitting with. In early Kafka the heartbeat was the poll loop - you were alive only if you polled. That coupled two unrelated things: a slow-but-healthy consumer (say, one doing a 30-second database write per batch) looked identical to a dead one, so it got kicked. So the heartbeat was moved to its own background thread. Now the heartbeat says "my process is running and my network is fine" on its own schedule, completely independent of how long your record processing takes.

That decoupling created the second clock. If heartbeats alone kept you in the group, a consumer that polled once and then hung forever in a bad while loop would keep its partitions and stall them indefinitely, because its heartbeat thread is still cheerfully beating. So Kafka added max.poll.interval.ms: a promise that you will come back and call poll() within this bound. Miss it, and the client library itself sends a LeaveGroup - the member fires the poll clock on itself.

Here are the knobs, what clock each belongs to, and their defaults - memorize the grouping, not the numbers:

The relationship the docs bury: heartbeat.interval.ms should be no more than about 1/3 of session.timeout.ms, so a member gets two or three chances to heartbeat before the coordinator gives up on a single dropped packet. And max.poll.interval.ms is essentially independent of the other two - it is a statement about your code's throughput, not your network.

The coordinator: where the leases are held

A rebalance needs a single arbiter - something that decides, authoritatively, who is in the group and what generation we're on. That arbiter is the group coordinator, and it is not a special server. It is just one of your regular brokers, chosen deterministically.

Kafka stores consumer offsets in an internal compacted topic called __consumer_offsets, which has 50 partitions by default. Your group is mapped to one of them by hash(group.id) % 50, and the broker that leads that partition is your group's coordinator. That's the whole election: no consensus round, just "whoever owns your slice of the offsets topic runs your group." When a consumer starts, it sends a FindCoordinator request to any broker, learns which broker owns its group, and talks to that one for everything after.

This has two consequences worth banking for debugging:

• If that broker restarts or the partition leader moves, your coordinator moves. Clients see a NotCoordinator/CoordinatorNotAvailable blip, re-run FindCoordinator, and reconnect - usually with a short rebalance. A rolling broker restart therefore ripples through consumer groups even though no consumer changed.
• The coordinator's memory of your group is the offsets topic. Group metadata (members, generation, committed offsets) is persisted there, so a coordinator that moves can reconstruct the group from the log.

The classic handshake: JoinGroup, then SyncGroup

Now the actual rebalance dance, in the "classic" protocol that every consumer used before Kafka 4.0 and that most still use today. A rebalance is a two-phase handshake orchestrated by the coordinator, and the group moves through a small state machine while it happens: Stable → PreparingRebalance → CompletingRebalance → Stable.

Classic rebalanceJoinGroup / SyncGroup, generation N → N+1

1. TriggerSomething changes membershipA member joins, leaves, or a clock fires. The coordinator moves the group to PreparingRebalance and stops it being Stable.
2. Phase 1 · JoinGroupAll members send their subscription and rejoinEach member sends what topics it wants. The coordinator waits up to the rebalance timeout (= max.poll.interval.ms) for stragglers, bumps the generation id, and picks ONE member as the group leader.
3. CoordinatorElect a leader, return the full member list to itThe leader is an ordinary consumer. The coordinator itself does not compute the assignment - it just hands the leader everyone's subscriptions.
4. Leader (client-side)Run the assignor, produce partition → member mapThe elected member runs the configured ConsumerPartitionAssignor (Range, RoundRobin, Sticky, CooperativeSticky) locally and decides who gets what.
5. Phase 2 · SyncGroupEveryone asks for their assignment; leader submits the mapAll members send SyncGroup. The leader's request carries the full assignment; the coordinator stores it and answers each member with just its slice. Group → CompletingRebalance → Stable.
6. MembersonPartitionsAssigned fires; consumption resumesEach member now owns its new partitions at generation N+1 and starts fetching. Any member still acting on generation N is fenced.

Assignment is computed on the client, by an elected leader - the coordinator just relays. Note how partitions are dark for the whole window: that's the eager stop-the-world we fix later.

Three details in that flow explain a surprising amount of real-world behavior:

• The assignment is computed on a client, not the broker. In the classic protocol the coordinator is a dumb relay; an elected member runs the assignment algorithm. This is why all members must agree on an assignor, and why a bad custom assignor can wedge a whole group.
• The generation id is a fence. Every completed rebalance bumps a monotonic generationId. Any request carrying a stale generation is rejected with IllegalGeneration, and that member must rejoin. This is how Kafka guarantees two members never both believe they own the same partition across a rebalance - the loser's offset commit simply fails. (That failure has a famous error message; it stars in Part 2.)
• The coordinator waits for the whole group at JoinGroup, up to the rebalance timeout, which for consumers equals max.poll.interval.ms. So a single slow member can hold the entire group in PreparingRebalance for minutes. One member's poll clock becomes everyone's stall.

Eager vs cooperative: the difference between a pause and a nudge

Now the single most important operational choice in the classic protocol, and the one most teams get wrong by inertia: how much of the group has to stop while the leases are re-cut.

Eager rebalancing: stop the world

The original protocol is eager. The rule is brutally simple: before a member rejoins at JoinGroup, it must revoke every partition it owns. All members, all partitions, released at once - then reassigned, then re-acquired. For the whole JoinGroup/SyncGroup window, nobody in the group is consuming anything, even the members whose partitions were never going to move.

RangeAssignor (the historical default), RoundRobinAssignor, and the original StickyAssignor all run eager. That trailing word "stop the world" is not hyperbole - it is literally what the assignor's RebalanceProtocol.EAGER contract requires, straight from the Kafka source: "a consumer to always revoke all its owned partitions before participating in a rebalance event."

Cooperative rebalancing: keep what you can, move only what must move

The cooperative protocol (Kafka's incremental cooperative rebalancing, via CooperativeStickyAssignor) changes the rule to: keep your partitions through the rebalance; only give up the specific ones that need to move to someone else. It does this with two lightweight rebalances instead of one heavy one:

1. First rebalance. Everyone rejoins but keeps consuming. The leader computes the new desired assignment and, by comparing it to the current one, works out exactly which partitions must change hands. Those - and only those - are marked for revocation.
2. Revoke the movers. The members losing those specific partitions call onPartitionsRevoked for just that subset.
3. Second rebalance. A follow-up rebalance hands the now-free partitions to their new owners.

The partitions that weren't moving never stopped. On a 100-partition group where one new member joins and needs ~10 partitions, eager pauses all 100; cooperative pauses ~10 and leaves 90 flowing.

The operational choiceone member joins a 100-partition, 10-member group

A rebalance is triggered11th member joins

how much stops while leases re-cut?

stop the worldEager (Range / RoundRobin / Sticky)All 10 members revoke all 100 partitions, rejoin, get reassigned. The whole group's throughput drops to zero for the JoinGroup + SyncGroup window. Simple, correct, and the reason deploys spike lag.

nudge, don't stopCooperative (CooperativeSticky)Only the ~9 partitions that move to the new member are revoked; the other ~91 keep flowing the entire time. Costs a second short rebalance, but no global pause. Preferred for anything latency-sensitive.

The assignors themselves

Which member gets which partition is the assignor's job. The ones you'll meet:

• RangeAssignor - default in the classic protocol. Assigns per topic: lays each topic's partitions in a row and cuts contiguous ranges to members. Simple, but co-locates partition 0 of every topic on the same member (handy for joins across topics) and skews badly when partition counts aren't divisible by member count.
• RoundRobinAssignor - deals all partitions across all topics round-robin. More even than range, but reshuffles heavily on membership change.
• StickyAssignor - aims for balance and minimal movement between rebalances, but still eager (revokes all, then tries to give most back).
• CooperativeStickyAssignor - the sticky logic, but cooperative: keeps partitions in place and moves the minimum. The modern default choice.

The word "sticky" is the tell: a sticky assignor tries to give you back the same partitions you had, so you keep your local state (RocksDB stores, in-flight buffers, warm caches) instead of rebuilding it every rebalance. On a stateful consumer that difference is enormous.

Static membership: making restarts free

There's a whole class of rebalances that shouldn't have to happen at all: a pod restarts during a rolling deploy, is gone for four seconds, and comes back as "the same" consumer - yet the classic protocol treats the restarted process as a brand-new member and rebalances twice (once when it leaves, once when it rejoins).

Static membership fixes this. Give each consumer a stable group.instance.id and it becomes a static member: the coordinator remembers that identity across disconnects. When a static member drops and reconnects within session.timeout.ms, the coordinator recognizes it, hands back the same assignment, and does not rebalance at all.

Restart behaviora consumer process restarts (deploy) and returns in 4s

Member disconnects, then reconnectssame host, new process

dynamic member or static member?

2 rebalancesDynamic (no group.instance.id)Coordinator saw an unknown member leave and an unknown member join. It rebalances on the leave (once the session clock fires) and again on the join. Multiply by every pod in a rolling deploy.

0 rebalancesStatic (group.instance.id set)Coordinator recognizes the returning identity within session.timeout.ms, returns the same partitions, and skips the rebalance entirely. Pair with a session timeout comfortably larger than your restart gap.

The trade you're making: a static member that truly dies isn't noticed until its (deliberately larger) session timeout expires, so genuine failures take longer to detect. Static membership is a bet that your restarts vastly outnumber your crashes - which, for a service on a rolling-deploy pipeline, they do.

The new protocol: KIP-848 moves the brains to the broker

Everything above is the classic protocol, and it has one structural flaw that no amount of tuning removes: the assignment is computed by a client, in a synchronous, stop-and-wait handshake where the whole group moves in lockstep and one slow member blocks everyone. Kafka 4.0 ships the replacement - the new consumer group protocol from KIP-848, enabled with group.protocol=consumer.

Two things change fundamentally:

1. The assignment moves server-side. The broker (coordinator) now computes the assignment itself, instead of relaying to an elected client leader. There is no more JoinGroup/SyncGroup election; there is no leader to run a buggy assignor.

2. Rebalancing becomes fully asynchronous and heartbeat-driven. There is a single RPC, ConsumerGroupHeartbeat, that does everything: it carries liveness, it carries the member's current assignment, and the broker's response carries the target assignment the member should move toward. Members reconcile toward that target incrementally and independently, acknowledging each step in their next heartbeat. No global barrier. No stop-the-world.

The mechanism that makes this safe is epochs. Instead of one generation id for the whole group, there's a group epoch (bumped on every membership or subscription change), a target-assignment epoch, and a member epoch per member. A member reconciles until its epoch matches the group's; if it ever acts on a stale epoch it gets a FencedMemberEpoch error and rejoins. Epochs are the new, finer-grained fence that replaces the coarse generation id.

KIP-848 reconciliationno JoinGroup/SyncGroup - just heartbeats carrying assignments

1. BrokerMembership changes → group epoch bumpsThe coordinator recomputes the target assignment server-side and bumps the group epoch. No election, no client leader.
2. HeartbeatMember learns its new target assignmentThe next ConsumerGroupHeartbeat response tells the member which partitions it should end up owning - not necessarily what it owns now.
3. Member (reconcile)Resolve topic IDs, commit offsets, revoke, assignThe member reconciles gradually: resolve topic names from metadata, auto-commit if enabled, call onPartitionsRevoked for what it must give up, onPartitionsAssigned for what it gains - all while still heartbeating.
4. Member (ack)Acknowledge the reconciled assignment via heartbeatThe member reports back the subset it actually reconciled. Its member epoch advances toward the group epoch.
5. BrokerEpochs converge → group is StableWhen every member's epoch matches the group epoch, reconciliation is done. Server group states: Empty → Assigning → Reconciling → Stable.

The member drives its own reconciliation and never stops heartbeating, so it stays in the group the whole time it's moving partitions. Contrast the classic timeline's dark window: here unaffected partitions never pause.

For operators, three practical shifts fall out of this design:

• session.timeout.ms and heartbeat.interval.ms become broker configs. In the new protocol they're controlled server-side (group.consumer.session.timeout.ms, group.consumer.heartbeat.interval.ms), not by each client. The session clock still exists - it just moved.
• The poll clock does not go away. max.poll.interval.ms is still enforced client-side, because the whole point of it - "are you actually calling poll()?" - is a property only the client can observe. The poll clock is the one that survives every protocol change, which is exactly why it's the one you end up debugging most.
• Rolling deploys and scale-ups stop being stop-the-world by default. Because reconciliation is incremental and per-member, the new protocol behaves like cooperative rebalancing without you configuring an assignor for it. Cooperative became the floor, not an opt-in.

The classic protocol isn't going away tomorrow - most running clusters are still on it, and the two interoperate during migration - but the direction is clear: the brains move to the broker, the handshake dissolves into heartbeats, and the coarse global generation becomes fine-grained per-member epochs. What doesn't change is the framing you came here for.

The throughline

Read the whole machine back to front and it collapses to something you can hold in one hand. A consumer group is a set of partition leases. Leases must be renewable, so every member is watched by two clocks: a session/heartbeat clock asking is the process alive? and a poll clock asking is it keeping up? A rebalance is what re-cuts the leases, and it is triggered by exactly four things - one of those two clocks firing, an intentional join/leave, or a partition-count change.

Everything else is implementation of that idea. The coordinator is just the broker that holds your leases. The JoinGroup/SyncGroup handshake, generations, eager vs cooperative, sticky assignors, static membership, and the KIP-848 epochs are all answers to how do we re-cut leases without two members ever thinking they own the same partition, and without stopping more of the group than we have to. The protocols evolved; the two clocks did not.

That's the payoff. When a group starts rebalancing in a loop at 3am, you don't reach for a random config. You ask one question - which clock fired? - and the answer tells you which knob to touch. Walking that question through nine real production failures, with the exact symptoms, log lines, and fixes for each, is Part 2: Debugging Kafka Consumer Group Rebalancing.

If you want the ground underneath this - how partitions, offsets, delivery semantics, and streaming ingestion actually fit together - that's the ingestion and transport curriculum, which builds the streaming model this post assumes from the first partition up.