Specified group generation id is not valid
Ahamad17 opened this issue · comments
Description
I started multiple stream app instances one after the other and when I start the producer while consumers were rebalancing, consumers were receiving the message but they are not able to commit the offset. Got the following exception. This exception is happening if I keep ConsumerConfig Guarantee to ProcessingGuarantee.EXACTLY_ONCE; It is happening when I spin up a new instance or kill an instance. Rebalancing is taking time and consumers get stuck in a "State transition from RUNNING to REBALANCING"
How to reproduce
I have a basic consumer stream that logs the message to the console
`
string inputTopic = "input";
var config = new StreamConfig<StringSerDes, StringSerDes>();
config.ApplicationId = "sample-streamiz-demo";
config.BootstrapServers = "localhost:9092";
config.AutoOffsetReset = AutoOffsetReset.Earliest;
config.Guarantee = ProcessingGuarantee.EXACTLY_ONCE;
StreamBuilder builder = new StreamBuilder();
var builderStream = builder.Stream<string, string>(inputTopic);
builderStream.Process(ProcessorBuilder.New<string, string>().Processor((record) =>
{
Console.WriteLine($"Process {record.Key} {record.Value}");
}).Build());
Topology t = builder.Build();
KafkaStream stream = new KafkaStream(t, config);
Console.CancelKeyPress += (o, e) => stream.Dispose();
await stream.StartAsync();`
Producer
`
try
{
_producer.BeginTransaction();
var taskResult = await _producer.ProduceAsync(topicName, new Message<string, T> { Key = Guid.NewGuid().ToString(), Value = kafkaEvent, Headers = getKafkaHeaders(headers) }, cancellationToken);
_producer.CommitTransaction();
return taskResult.TopicPartitionOffset.ToString();
}
catch (Exception ex)
{
_producer.AbortTransaction();
return string.Empty;
}`
Streamiz.Kafka.Net nuget version = 1.4.2
Consumer logs.txt
Broker logs.txt
Attaching both app and broker logs below
Help asked in confluent-kafka-dotnet repository : confluentinc/confluent-kafka-dotnet#2008