Add `static membership` Kafka feature to clients
andsel opened this issue · comments
The static membership concept is proposed and explained in KIP-345.
When a client of a consumer group reconnects to a cluster it triggers a re-balance, to assign partition to consumers and this consumer is assigned an id named member.id which is randomly generated.
If a client is member of a consumer groups, and reconnects frequently is mean it triggers frequent re-balances.
To avoid this re-assignment of consumer to partitions during re-balancing, a static member.id could be passed by the consumer, to uniquely identify it inside the consumer group and in stored on both sides (broker and consumer). In this way the static group.instance.id avoids re-balances in case of continuous disconnections and re-connections.
Also with Static Membership however, re-balances could happen when a new member joins the group or a leader re-joins, for a full list of cases where re-balances happen also with static membership, please check KIP-345
What happens if group.instance.id is not static?
From paragraph Join Group Logic Change in KIP-345
The edge case is that if we could have members with the same `group.instance.id` (for example mis-configured
instances with a valid `member.id` but added a used `group.instance.id` on runtime).
When `group.instance.id` has duplicates, we could refuse join request from members with an outdated
`member.id`...[omissis]...In an edge case where the client hits this exception in the response,
it is suggesting that some other consumer takes its spot. The client should immediately fail itself
to inform end user that there is a configuration bug which is generating duplicate consumers
with same identity
So in that case the client would fail the joining request.