Scaling Kafka Consumers Without Losing Your Weekend

Consumer lag creeps up quietly. One day your topic is keeping up fine, and a few weeks later you're paging someone at 2am because a partition fell behind.

What actually happened

A single partition kept lagging while the other five stayed healthy. The consumer group looked balanced on paper, but one instance was doing noticeably more work than the rest because of uneven key distribution.

The fix

Three changes, in order of impact:

1. Moved from a hash-based partition key to one that spread load more evenly across the keyspace.
2. Tuned max.poll.records down so a single slow batch couldn't stall the whole partition.
3. Added a lag-based alert instead of only alerting on consumer group failures, so we'd notice the trend before it became an incident.

Takeaway

Rebalancing is rarely the actual root cause — it's usually the symptom of uneven work distribution upstream. Fix the skew before touching consumer group settings.