microsoft / CSharpClient-for-Kafka

.Net implementation of the Apache Kafka Protocol that provides basic functionality through Producer/Consumer classes. The project also offers balanced consumer implementation.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

An existing consumer can become locked if consumed offset is too low

ducas opened this issue · comments

Use Case

As an consumer in an existing group
I want to know when my offset is out of range
So that I can make a decision about how to proceed

Description

As described in issue #39, a balanced consumer will stop receiving messages if its committed offset is too far behind the start of the topic. This line seems to be causing this problem - https://github.com/Microsoft/Kafkanet/blob/master/src/KafkaNET.Library/Consumers/FetcherRunnable.cs#L83

IEnumerable fetchablePartitionTopicInfos = _partitionTopicInfos.Where(pti => pti.NextRequestOffset - pti.ConsumeOffset < _fetchBufferLength);

If the consume offset is too low then the FetcherRunnable will consider it not fetchable and not issue any fetches...

We need to either present this as an issue to the consuming library or move on our merry way by resetting it to whatever the value of auto.offset.reset is.

Repro Steps

  1. Create a balanced consumer with a max fetch batch size of 1000 (default) and start consuming messages from a topic with a short retention period
  2. Stop the consumer, push a lot of messages in and wait until the retention period has passed and the consumed offset is more than 1000 behind the start of the topic
  3. Start the consumer
  4. Observe nothing happening...