org.apache.kafka.common.errors.TimeoutException: Expiring record(s) for eventhub-partition

[deepsandeepme]

Description

A series of pod restarts followed by large number of exceptions in the logback errors have been observed. The latencies during these occurrences are also quite significant. During peak hours we are seeing a lot of timeout exceptions in the format as below:

org.apache.kafka.common.errors.TimeoutException: Expiring 5 record(s) for eventhub-partitionId:120000 ms has passed since batch creation

We are seeing this exception in 10000s of count during the peak hours. Consequently, we do see our kubernetes pod restarts because of these exceptions and thus causing the latency. We are currently using the Microsoft recommended configurations and using a dedicated event hub cluster with 12 CU.

ALA logs show exceptions thrown during this time duration. All the logs have the same Timeout Exception, across various event hub partitions.

We have the following configured in our system now:

spring.cloud.stream.kafka.binder.configuration.metadata.max.age.ms=180000
spring.cloud.stream.kafka.binder.configuration.connections.max.idle.ms=180000
spring.cloud.stream.kafka.binder.configuration.max.request.size=1000000
spring.cloud.stream.kafka.binder.configuration.retries=0
spring.cloud.stream.kafka.binder.configuration.request.timeout.ms=60000
spring.cloud.stream.kafka.binder.configuration.linger.ms=150
spring.cloud.stream.kafka.binder.configuration.delivery.timeout.ms=120000
spring.cloud.stream.kafka.binder.configuration.max.poll.records=500

How to reproduce

These errors are happening for past 2 weeks now and we see this during peak hours, followed by multiple pods restarting and causing latency.

We tried to go through your recommendation provided in the following stackoverflow link, but we are already using the recommended configurations.

https://stackoverflow.com/questions/58010247/azure-eventhub-kafka-org-apache-kafka-common-errors-timeoutexception-for-some-of

Has it worked previously?

Our system was working successfully without any errors until 5-April-2023. After that we started seeing this error.

Checklist

IMPORTANT: We will close issues where the checklist has not been completed or where adequate information has not been provided.

Please provide the relevant information for the following items:

• SDK (include version info): 17

• Sample you're having trouble with: NA

• If using Apache Kafka Java clients or a framework that uses Apache Kafka Java clients, version: <REPLACE with e.g., 1.1.0>

• Kafka client configuration: provided above

• Namespace and EventHub/topic name: Dedicated tier, Cannot share the details in the open forum

• Consumer or producer failure: producer failure

• Timestamps in UTC `Since 05-April on peak EMEA hours

• group.id or client.id <REPLACE with e.g., group.id=cg-name>

• Logs provided (with debug-level logging enabled if possible, e.g. log4j.rootLogger=DEBUG) or exception call stack:
  org.apache.kafka.common.errors.TimeoutException: Expiring 9 record(s) for<<EVENTHUB_NAME>>-83:120000 ms has passed since batch creation

• Standalone repro : NA

• Operating system: 'linux-version unknown'

• Critical issue

If this is a question on basic functionality, please verify the following:

• Port 9093 should not be blocked by firewall ("broker cannot be found" errors)
• Pinging FQDN should return cluster DNS resolution (e.g. $ ping namespace.servicebus.windows.net returns ~ ns-eh2-prod-am3-516.cloudapp.net [13.69.64.0])
• Namespace should be either Standard or Dedicated tier, not Basic (TopicAuthorization errors)