An existing consumer can become locked if consumed offset is too low
ducas opened this issue · comments
Use Case
As an consumer in an existing group
I want to know when my offset is out of range
So that I can make a decision about how to proceed
Description
As described in issue #39, a balanced consumer will stop receiving messages if its committed offset is too far behind the start of the topic. This line seems to be causing this problem - https://github.com/Microsoft/Kafkanet/blob/master/src/KafkaNET.Library/Consumers/FetcherRunnable.cs#L83
IEnumerable fetchablePartitionTopicInfos = _partitionTopicInfos.Where(pti => pti.NextRequestOffset - pti.ConsumeOffset < _fetchBufferLength);
If the consume offset is too low then the FetcherRunnable will consider it not fetchable and not issue any fetches...
We need to either present this as an issue to the consuming library or move on our merry way by resetting it to whatever the value of auto.offset.reset is.
Repro Steps
- Create a balanced consumer with a max fetch batch size of 1000 (default) and start consuming messages from a topic with a short retention period
- Stop the consumer, push a lot of messages in and wait until the retention period has passed and the consumed offset is more than 1000 behind the start of the topic
- Start the consumer
- Observe nothing happening...