FailedToSendMessageException when RequiredAcks != 0

[Honza Brestan]

Hi everyone,

I'm using CSharpClient-for-Kafka NuGet 1.0.47 against Kafka 0.9.0.1 to implement a simple producer.

Every time I try to send a message with non-default ProducerConfiguration.RequiredAcks, the client fails after 3 attempts with FailedToSendMessageException.
Interesting fact is that the message is actually delivered to the broker three times, and even passed to the consumers three times. When I don't use any acknowledgements, sending message works perfectly fine, but for my use case I need delivery guarantees, so RequiredAcks=0 is not really an option.

I've tried using the Kafka built-in kafka-console-producer.sh and it works as expected even with --request-required-acks set to 1 or -1 - the message is delivered once and with no error.

This leads me to believe there's either an issue with the Producer (not processing acks correctly?) or the way I'm using it.

The topic has 5 partitions on 2 brokers, replication factor 1.

My code to reproduce:

var brokers = new[] { ... };    
var config = new ProducerConfiguration(brokers){RequiredAcks=1};
var producer = new Producer(config);
var message= new Message(Encoding.UTF8.GetBytes("Hi from C#"));
var batch = new ProducerData<string, Message>("test", "key", message);
producer.Send(batch);

The exception thrown on producer.Send:

Failed to send messages after 3 tries. FailedProducerDatas not empty. Success Count:0 Failed Count: 1.
================Failed sent message key: Topic: test Key: key
================Failed Detail: Broker:0,Topic:test PartitionID:1,Error:UnknownCode Offset:-1

Stack trace:

   at Kafka.Client.Producers.DefaultCallbackHandler`2.Handle(IEnumerable`1 events)
   at Kafka.Client.Producers.Producer`2.Send(IEnumerable`1 data)
   at Kafka.Client.Producers.Producer`2.Send(ProducerData`2 data)

kafka-console-consumer.sh output after a single call:

Hi from C#
Hi from C#
Hi from C#

Thank you for any advice!

[Ashish Dhyani]

I have the same issue for RequiredAcks=1, messages are delivering thrice.
But for RequiredAcks=0, its randomly failing to send the message. I have sent 137 messages. Kafka received only 66 and rest of them are failed/undelivered. Can anyone look into this issue ?

@jbrestan , in your case you can do workaround:
var config = new ProducerConfiguration(brokers){RequiredAcks=1, ProducerRetries = 1};

[Soumyajit Sahu]

This library works only with Kafka 0.8. Please use the https://github.com/ah-/rdkafka-dotnet for 0.9 or higher.

[Honza Brestan]

@ashishdhyani I'm not sure I remember the exact behavior, but I think even with that workaround I would always get an exception on the producer and thus I wouldn't be able to recognize whether it "normally" threw the exception, or whether it actually failed to deliver the message - which I do care about.

@soumyajit-sahu Thanks, I ended up using https://github.com/gigya/KafkaNetClient and it also works (even against 0.10 cluster, at least in backwards compatible mode with the older protocol)

[Ashish Dhyani]

Hi @jbrestan, @soumyajit-sahu, after checking the kafka log I found that ClientId is required for the correct usage.
Message drop or duplicate message fixed after I provided the ClientId as unique id.

below code works

var config = new ProducerConfiguration(brokers){ RequiredAcks=1, ClientId = Guid.NewGuid().ToString() };