criteo / kafka-sharp

Which leads to exception on deserialization response in case when Compatibility is set to V0_10_1.

How to reproduce:

Use 10.1 broker
Set Compatibility to V0_10_1
Call ClusterClient.Subscribe for unexisting group without offsets and existing topic

In my case message consumption is not starting and log is full of exceptions like

ERROR A response could not be decoded for the node [Id:3 Host:localhost Port:9092]: System.OverflowException: Array dimensions exceeded supported range.
   at Kafka.Protocol.Basics.DeserializeArray[TData](ReusableMemoryStream stream, Func`2 dataDeserializer)
   at Kafka.Protocol.OffsetPartitionResponse.Deserialize(ReusableMemoryStream stream, Object noextra, ApiVersion version)
   at Kafka.Protocol.TopicData`1.Deserialize(ReusableMemoryStream stream, Object extra, ApiVersion version)
   at Kafka.Protocol.Basics.DeserializeArrayExtra[TData](ReusableMemoryStream stream, Object extra, ApiVersion version)
   at Kafka.Cluster.Node.Serialization.DeserializeCommonResponse[TPartitionResponse](Int32 correlationId, ReusableMemoryStream data, ApiVersion version)
   at Kafka.Cluster.Node.ProcessOffsetResponse(Int32 correlationId, ReusableMemoryStream responseData, IBatchByTopic`1 originalRequest)

Thank you

Are there any messages in your topic?

@sdanzan
Yes, topic is not empty

What do you mean by "TopicRequest is always using ApiVersion.V0". TopicRequest is not an API (I think you mean either Metadata, Offsets or ListOffsets), and here the stack trace seems to suggest that brokers are returning an empty list of offsets when we try to get the current begin/end offsets for the partitions (it is a bug, this case should not be treated as an exception, but it does not break the client and it should get back on its feet whenever the brokers returns a non empty list).

Sorry if I confused terminology.

Well, I might be wrong with this, but look what I've found:

Everything works for Compatibility V0_8_2
In debugging session I saw wierd values during response deserialization. It doesn't related to empty offsets list (it is empty actually), but looked exactly like V0 response. In this particular case array length was below zero.

kafka-sharp/kafka-sharp/kafka-sharp/Protocol/Basics.cs

Line 268 in 323b1c9

var array = new TData[count];
I've found hardcoded API version here

kafka-sharp/kafka-sharp/kafka-sharp/Protocol/TopicRequest.cs

Line 16 in 323b1c9

return CommonRequest.Serialize(target, this, correlationId, clientId, Basics.ApiKey.MetadataRequest, Basics.ApiVersion.V0, null);

(Ok, it can be unrelated, I don't actually sure)

Oh, TopicRequest.cs is just a convenient wrapper over the Metadata API when requiring informations for one topic instead of all. Metadadata API has only version 0.

Currently deserialization of empty arrays is badly handled, however empty arrays are never returned except when listing offsets on empty topics (I probably overlooked this test case). I will correct this bug to avoid logging a bazillion of thrown exceptions, however when it happens, the underlying state machine is not broken. I must understand now why your brokers always return empty arrays when the client is in v10 compat (I've definitely tested reading from 0.10.1 brokers with clients in v10 compat mode and it worked).

Also, are you using some kind of compression?

the underlying state machine is not broken

Ok, it's not broken but it's not fetching messages

Correct me If I wrong, to fetch messages, consumer must know list of partitions and their offsets for correctly sending fetch requests. Exception in OffsetPartitionResponse deserialization makes this impossible.

This code is reproducing behaviour. Please note, that group_id has no commited offsets and my topic has 5 partitions and many many messages.


static void Main(string[] args)
        {
            var brokerList = "localhost:9092";

            var configuration = new Configuration
            {
                Seeds = brokerList,            
                Compatibility = Compatibility.V0_10_1
            };

            var clusterClient = new ClusterClient(configuration, new ConsoleLogger());

            var groupConfig = new ConsumerGroupConfiguration
            {
                DefaultOffsetToReadFrom = Offset.Lastest,
                AutoCommitEveryMs = -1
            };

            clusterClient.MessageReceived += x => Console.WriteLine($"{x.Partition} : {x.Offset}");

            clusterClient.Subscribe("NewGroup", new[] { "ExistingTopicWithMessages" }, groupConfig);

            Console.ReadKey();
        }

UPD: we use Gzip

I must understand now why your brokers always return empty arrays

I guess that's because request was as V0 and response in V0 too.

Ok, I found the problem. There is a mismatch between the Offset API (api key 2) described in https://kafka.apache.org/protocol and https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol. The latter is incorrect but it's generally the one I follow since it's referred by the former and is the only doc with the full message set format description. I've probably actually never tested the case where a consumer starts with no saved offsets against v0.10.1+ brokers.

I will merge the patches later this week end (or maybe next monday). The workaround meanwhile is to use 0.8 compat mode, at least when consuming (it's a generally negligible loss of performances when used against 0.10 brokers).

Thank you

Should be fixed now.

Ok, I'l check later