Kafka Consumer Group Rebalancing: Why Your Consumers Stall and How to Stop It

The pager said consumer lag on orders.enriched had crossed two million records. The consumers were up. CPU was flat. The broker was healthy. Logs showed the same three lines cycling every ninety seconds: Revoke previously assigned partitions, (Re-)joining group, Successfully joined group with generation 4417.

Generation 4417. The group had rebalanced four thousand times since deploy. Between each rebalance it processed for maybe forty seconds, then stopped dead for another thirty while every consumer in the group handed back every partition and waited for a new assignment. Throughput was not limited by the consumers. It was limited by how much of each minute they were allowed to work.

That failure shape is the single most common Kafka incident I have been called into, and it is almost never a broker problem. It is a poll loop that got slower than someone’s default config assumed, combined with an assignment strategy that punishes the entire group for one member’s tardiness. This post is the protocol, the timeouts, and the configuration I now set on day one so the incident does not happen. If partitions and offsets are still fuzzy, start with Kafka fundamentals.

What a rebalance actually is

A consumer group is a set of consumers sharing the partitions of one or more topics. Every partition goes to exactly one member. The mapping from partitions to members is the assignment, and recomputing it is a rebalance.

One broker per group acts as the group coordinator. It tracks membership, holds the committed offsets in __consumer_offsets, and increments a generation counter every time the group changes. A rebalance is triggered by any of these:

• A consumer joins (deploy, scale-up, restart).
• A consumer leaves cleanly (close() sends a LeaveGroup).
• A consumer stops heartbeating for session.timeout.ms.
• A consumer fails to call poll() within max.poll.interval.ms.
• Partition count on a subscribed topic changes.
• The coordinator broker itself moves.

The first member to rejoin becomes the group leader, and the leader computes the assignment on the client side using the configured partition.assignment.strategy. The coordinator just distributes it. That detail matters: assignment strategy is a client config, and a group running mixed strategies across a rolling deploy will negotiate down to whatever all members share.

sequenceDiagram
  participant C1 as "Consumer 1"
  participant C2 as "Consumer 2"
  participant GC as "Group coordinator"
  C1->>GC: Heartbeat
  C2->>GC: Heartbeat
  Note over C2: "Poll loop takes 6 minutes"
  GC->>C1: "Rebalance in progress"
  C1->>GC: JoinGroup
  GC->>C1: "You are leader, gen 42"
  C1->>GC: "SyncGroup with assignment"
  GC->>C1: "Partitions 0-3"
  C2->>GC: "Commit fails: unknown member"

One slow consumer stops the group. The commit it tried to make on return is rejected as a zombie.

Eager versus cooperative sticky

Until RangeAssignor and RoundRobinAssignor were joined by CooperativeStickyAssignor, every rebalance used the eager protocol: all members revoke all partitions, then everyone rejoins, then a new assignment goes out. The group processes zero records for the whole duration. This is the stop-the-world pause that turns a routine deploy into a lag graph shaped like a staircase.

Cooperative rebalancing (incremental) changes the deal. Members keep the partitions they are keeping. The leader computes the new assignment, revokes only the partitions that must move, and a second short round hands those to their new owner. A twelve-member group scaling to thirteen moves roughly one partition per member instead of pausing all twelve.

The config is one line, and it is the highest-value line in this post:

# Client-side. Every consumer in the group needs it.
partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor

The migration is the part people get wrong. You cannot flip all consumers at once if you care about correctness, because a group cannot mix eager and cooperative in a single generation. The supported path is two deploys:

1. Deploy with the list CooperativeStickyAssignor,RangeAssignor. The group keeps negotiating RangeAssignor (the only strategy all members share) until every pod has the new config.
2. Deploy again with CooperativeStickyAssignor alone. Now the common strategy is cooperative, and the group upgrades on the next rebalance.

Skip step one and you get an assignment exception loop that looks identical to the incident you were trying to fix. I have done this at 2 a.m. Do not.

The four timeouts, and what each one detects

These get tuned as a group by people who think they all mean “how long before Kafka gives up.” They detect different failures.

heartbeat.interval.ms (default 3000). How often the background heartbeat thread pings the coordinator. It is a separate thread from your poll loop. Set it to roughly one third of session.timeout.ms.

session.timeout.ms (default 45000 in modern clients). How long the coordinator waits without a heartbeat before declaring the member dead. This detects crashed consumers, network partitions, and long GC pauses. Lower means faster failure detection and more false positives.

max.poll.interval.ms (default 300000). How long between two calls to poll() before the consumer removes itself from the group. This detects slow consumers. Because heartbeats are on their own thread, a consumer stuck in a six-minute batch write keeps heartbeating happily while blowing this limit.

max.poll.records (default 500). The lever that actually controls poll duration. Your worst-case poll time is roughly max.poll.records multiplied by the p99 per-record processing time, plus any fixed batch flush cost.

The arithmetic is the whole game:

max.poll.interval.ms  >  max.poll.records x p99_per_record_ms x safety_factor

For a consumer writing to a warehouse at 40 ms per record at p99, 500 records is 20 seconds of work — fine against a 300-second limit. For a consumer doing a synchronous HTTP enrichment call at 250 ms p99, 500 records is 125 seconds, and a downstream slowdown to 700 ms puts you at 350 seconds and a self-eviction. That is the orders.enriched incident, exactly.

The fix is almost always to lower max.poll.records, not to raise max.poll.interval.ms. Raising the interval means a genuinely hung consumer holds its partitions for ten minutes before anyone notices. Lowering the record count keeps the loop responsive.

# confluent-kafka-python. The config I start from for a sink
# consumer that writes batches to a warehouse.
from confluent_kafka import Consumer, KafkaException

conf = {
    "bootstrap.servers": "b-1.msk.internal:9092",
    "group.id": "orders-enricher",
    "partition.assignment.strategy": "cooperative-sticky",
    # Detect dead members in 45s, heartbeat every 15s.
    "session.timeout.ms": 45000,
    "heartbeat.interval.ms": 15000,
    # 120 records x ~250ms p99 = 30s of work per loop, well under 120s.
    "max.poll.interval.ms": 120000,
    "max.poll.records": 120,
    # We commit explicitly after the sink write succeeds.
    "enable.auto.commit": False,
    "auto.offset.reset": "earliest",
    # Survive rolling restarts without a rebalance.
    "group.instance.id": os.environ["POD_NAME"],
}
consumer = Consumer(conf)

Static membership removes the deploy rebalance

group.instance.id gives a consumer a stable identity. When a consumer with a static ID disconnects, the coordinator does not immediately rebalance. It holds that member’s partitions until either the member returns or session.timeout.ms expires.

For a Kubernetes StatefulSet whose pods restart in eight seconds, setting group.instance.id to the pod name and session.timeout.ms to 45 seconds means a rolling restart of a twelve-pod deployment causes zero rebalances. The partitions sit idle for a few seconds per pod and then resume with the same owner, warm caches and all.

Two rules come with it. First, the ID must be genuinely stable — a random UUID per boot is worse than nothing, because now dead members linger for the full session timeout. StatefulSet ordinals work; Deployment pod names do not. Second, a static member that truly dies keeps its partitions unserved for the whole session timeout, so do not set that to ten minutes because it made your deploy graph pretty.

Instrument the rebalance, do not guess at it

Every consumer I run registers a rebalance listener that logs and counts. Without it, “the group is rebalancing” is a hypothesis you support with log greps.

// Cooperative protocol: onPartitionsRevoked gets only the partitions
// actually moving away, and it can be empty on most rebalances.
consumer.subscribe(List.of("orders.enriched"), new ConsumerRebalanceListener() {
    @Override
    public void onPartitionsRevoked(Collection<TopicPartition> revoked) {
        if (revoked.isEmpty()) return;
        // Flush the sink and commit BEFORE we lose ownership.
        sink.flush();
        consumer.commitSync(offsetsFor(revoked));
        meter.counter("kafka.rebalance.revoked").increment(revoked.size());
        log.warn("revoked {} partitions, generation={}",
                 revoked.size(), consumer.groupMetadata().generationId());
    }

    @Override
    public void onPartitionsAssigned(Collection<TopicPartition> assigned) {
        meter.counter("kafka.rebalance.assigned").increment(assigned.size());
    }

    @Override
    public void onPartitionsLost(Collection<TopicPartition> lost) {
        // We were evicted. Someone else may already own these.
        // Do NOT commit here - the commit will be rejected anyway.
        sink.discardPending();
        meter.counter("kafka.rebalance.lost").increment(lost.size());
    }
});

onPartitionsLost is the one people omit. It fires when the consumer was fenced out (max poll exceeded, session expired) and the partitions are already gone. Committing there produces the CommitFailedException that fills the logs during a rebalance storm. Flushing a sink there produces duplicate writes from a consumer that no longer owns the data — which is why sinks need idempotency regardless of your delivery semantics, a point I make at length in the exactly-once post.

The three metrics I alert on:

• rebalance-rate-per-hour above 2 for a steady group. Deploys aside, a healthy group rebalances approximately never.
• rebalance-latency-max above 30 seconds. That is your stop-the-world budget.
• last-poll-seconds-ago approaching max.poll.interval.ms — a leading indicator, unlike lag, which is a lagging one.

Sizing partitions and members

A partition is the unit of parallelism. Members beyond the partition count sit idle holding nothing, which is a legitimate hot-standby pattern but a waste if you meant to scale throughput.

My defaults: partition count at two to four times the expected peak consumer count, so you can scale out twice without a repartition. Never let a member own more than about twenty partitions if each one has independent buffered state, because the revocation flush at rebalance time becomes its own poll-interval violation.

Repartitioning a topic is not a free scaling operation either. It changes key-to-partition mapping for all future writes, which breaks per-key ordering across the boundary and quietly corrupts any downstream aggregation that assumes a key’s history lives in one partition. Size up front, or plan a topic migration with a dual-write window the way you would a contract version bump.

flowchart TD
  slow["Poll loop slower than budget"] --> evict["Member self-evicts"]
  evict --> reb["Group rebalance"]
  reb --> cold["New owner starts cold"]
  cold --> slower["First polls even slower"]
  slower --> slow
  fix1["Lower max.poll.records"] --> slow
  fix2["Cooperative sticky"] --> reb
  fix3["group.instance.id"] --> reb

The storm is a feedback loop. Break it at the poll budget, not at the timeout.

Pitfalls

Raising max.poll.interval.ms to 30 minutes. It stops the symptom and hides a genuinely stuck consumer for half an hour. Lower max.poll.records instead, and only raise the interval when you have a real batch operation with a known ceiling.

Doing blocking I/O inside the poll loop with default settings. An HTTP call, a SELECT against an overloaded Postgres, an S3 PUT with retries — any of these can go from 50 ms to 5 seconds under load, and 500 records times 5 seconds is way past every default.

Flipping to cooperative sticky in one deploy. Mixed protocols in one group throw on assignment. Two deploys, always.

Random group.instance.id values. A UUID per process start means every restart leaves a ghost member holding partitions for the full session timeout. Worse than not using static membership.

Committing in onPartitionsLost. The commit is rejected, the exception is logged as an error, and someone spends a day chasing a message that is just the protocol telling you that you lost.

Treating consumer lag as the primary alarm. Lag tells you that you are behind. Rebalance rate and time-since-last-poll tell you why, and they move first.

FAQ

Does cooperative sticky eliminate rebalance pauses?

It eliminates the group-wide pause, not the rebalance. Members that keep their partitions keep processing throughout. Only the partitions that genuinely change owners see a gap, and that gap is usually a second or two rather than the full protocol round trip.

Why do I see rebalances with no deploys and no failures?

Check for a consumer whose poll() occasionally exceeds the interval — a nightly compaction, a cache refresh, a retry storm against a downstream service. Also check whether the coordinator broker is being restarted by your cloud provider’s patching window; a coordinator move triggers a group-wide rejoin.

Should I set session.timeout.ms lower for faster failover?

Rarely. Below about 20 seconds you start evicting healthy consumers during GC pauses and brief network blips, and each false eviction costs more availability than it saves. I keep 45 seconds and rely on liveness probes to kill genuinely dead pods faster than the timeout would.

Do I need static membership if I already use cooperative sticky?

They solve overlapping but different problems. Cooperative makes each rebalance cheaper; static membership makes routine restarts not rebalance at all. On Kubernetes with frequent deploys I use both, and the deploy-time lag spike disappears entirely.

How does this interact with Kafka Streams?

Kafka Streams has used a sticky task assignor for years and adds warmup replicas so state stores migrate in the background before the task moves. The same poll-budget arithmetic applies, and max.poll.records is still the lever, but you also need to watch num.standby.replicas if state restore time is your real pause.

Is KIP-848 going to make this obsolete?

The next-generation consumer rebalance protocol moves assignment computation to the broker and removes the global sync barrier, which does make several of these failure modes structurally impossible. It is worth adopting when your cluster and clients both support it. The poll-budget discipline in this post still applies, because a consumer that cannot keep up is a consumer that cannot keep up regardless of protocol.

What this means for your pipelines

Start by measuring one number: the p99 duration of a single poll()-to-poll() cycle in each consumer group you run. Almost nobody has this on a dashboard, and every timeout in this post is a derivative of it. Once you have it, set max.poll.records so the worst-case loop finishes in under a quarter of max.poll.interval.ms, and you have bought yourself a 4x degradation budget before anything gets evicted.

Then make the two configuration changes that cost nothing: cooperative sticky assignment across two deploys, and group.instance.id bound to a genuinely stable pod identity. Together they remove the deploy-time rebalance and shrink the remaining ones from group-wide stalls to per-partition handoffs. For most teams this is the difference between a lag graph that sawtooths on every release and one that stays flat.

Finally, accept that rebalances will still happen, and design the consumer so one is boring. Flush and commit in onPartitionsRevoked, discard and move on in onPartitionsLost, and make the sink idempotent so a duplicate batch from a fenced consumer is a no-op rather than a data quality incident. A rebalance you can survive without thinking about it is the actual goal; the tuning above is just how you get the frequency down to the point where surviving them is cheap.