tulios / kafkajs

A modern Apache Kafka client for node.js

Home Page:https://kafka.js.org

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

RoundRobin Assignor doesn't acknowledge Member subscribed topics in assignment

edeesis opened this issue · comments

Describe the bug
Consumers in a Consumer Groups can subscribe to different topics. When a group rebalances, the elected leader uses Assigner#assign to determine what topic-partitions to assign to which group members.

However, the current implementation only looks at the leader's subscribed topics when it does this assignment, rather than the subscribed topics of each of the members.

Here's an example:

Give 2 topics, Topic A with 2 partitions, Topic B with 2 partitions

Consumer Group ID (Leader?) Subscribed Topics Client ID (used to generate Group Member ID) Assigned Topic Partitions
group-1 (Leader) Topic A group-1-1 Topic A Partition 1
group-1 Topic B group-1-2 No assignments
group-1 Topic A group-1-3 No Assignments
group-1 Topic B group-1-0 Topic A Partition 0

The client IDs are just to show that there's a alphanumeric sort applied to member IDs to keep partition assignment relatively stable across rebalances.

To Reproduce

  1. Create two topics, each with 1 partition, called topic-a, and topic-b
  2. Run two consumers, with the same group ID. The first consumer should subscribe to topic-a and the client ID of "client-id-1" and the second should subscribe to topic-b with the client ID "client-id-0". The first consumer should become the leader when it starts up.
  3. Use the KafkaAdmin describeGroup to list the group members and assignments. You should see the second consumer assigned topic-a, even though its subscribed to topic-b.

Expected behavior
The round robin assigner should assign topic-partitions only to members that are subscribed to those topics.

Observed behavior
The round robin assigner assigns topics to consumers that they aren't subscribed to.

Environment:

  • OS: Mac OS 14.4.1 (in Docker)
  • KafkaJS version: 2.2.4
  • Kafka version: 7.3.1-ccs (Confluent)
  • NodeJS version: 20.10.0

Additional context
The Java RoundRobinAssignor more intelligently maps partition assignments via subscribed topics as described in the JavaDocs