Giters

confluentinc / librdkafka

The Apache Kafka C/C++ library

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Issue with Confluent Kafka Python and MSK: "metadata information unknown"

tuliolemes opened this issue · comments

commented

Hi everyone,
I hope you're all doing well.

Description

I'm encountering an issue while trying to process a message using Confluent Kafka Python and Amazon MSK.
Below is the relevant code snippet:

How to reproduce

    def delivery_report(self, err, msg):
        if err is not None:
            print("Message delivery failed: {}".format(err))
        else:
            print("Message delivered to {} [{}]".format(msg.topic(), msg.partition()))

    def process_message(self, message: str):
        print("DEBUG: Polling")
        self.producer.poll(0)
        self.producer.produce(self.kafka_config.topic, value=message.encode("utf-8"), on_delivery=self.delivery_report)

        print("DEBUG: Flushing {} records...".format(len(self.producer)))
        self.producer.flush()
        print("DEBUG: Flushing 2")

Checklist

  • librdkafka version (release number or git tag): librdkafka v2.3.0
  • Apache Kafka version: I believe it's 3.5.1 but I'm no sure, I'm using MSK Serverless
  • librdkafka client configuration:
%7\|1706584609.068\|CONF\|ip-172-20-156-228.us-east-2.compute.internal#producer-2\| [thrd:app]: Client configuration:
--
%7\|1706584609.068\|CONF\|ip-172-20-156-228.us-east-2.compute.internal#producer-2\| [thrd:app]:   client.id = ip-172-20-156-228.us-east-2.compute.internal
%7\|1706584609.068\|CONF\|ip-172-20-156-228.us-east-2.compute.internal#producer-2\| [thrd:app]:   client.software.name = confluent-kafka-python
%7\|1706584609.068\|CONF\|ip-172-20-156-228.us-east-2.compute.internal#producer-2\| [thrd:app]:   client.software.version = 2.3.0-rdkafka-2.3.0
%7\|1706584609.068\|CONF\|ip-172-20-156-228.us-east-2.compute.internal#producer-2\| [thrd:app]:   metadata.broker.list = boot-[...]:9098
%7\|1706584609.068\|CONF\|ip-172-20-156-228.us-east-2.compute.internal#producer-2\| [thrd:app]:   debug = generic,broker,topic,metadata,feature,queue,msg,protocol,cgrp,security,fetch,interceptor,plugin,consumer,admin,eos,mock,assignor,conf,all
%7\|1706584609.068\|CONF\|ip-172-20-156-228.us-east-2.compute.internal#producer-2\| [thrd:app]:   error_cb = 0x7fb55d89d110
%7\|1706584609.068\|CONF\|ip-172-20-156-228.us-east-2.compute.internal#producer-2\| [thrd:app]:   opaque = 0x7fb55cc8d4e0
%7\|1706584609.068\|CONF\|ip-172-20-156-228.us-east-2.compute.internal#producer-2\| [thrd:app]:   security.protocol = sasl_ssl
%7\|1706584609.068\|CONF\|ip-172-20-156-228.us-east-2.compute.internal#producer-2\| [thrd:app]:   sasl.mechanisms = OAUTHBEARER
%7\|1706584609.068\|CONF\|ip-172-20-156-228.us-east-2.compute.internal#producer-2\| [thrd:app]:   sasl.oauthbearer.config = [redacted]
%7\|1706584609.068\|CONF\|ip-172-20-156-228.us-east-2.compute.internal#producer-2\| [thrd:app]:   dr_msg_cb = 0x7fb55d89a9e0
  • Operating system: I believe Linux 2.1, since it's running in MSK Serverless
  • Provide logs (with debug=.. as necessary) from librdkafka:
DEBUG: Polling
--
DEBUG: Flushing 1 records...
%7\|1706584612.249\|TOPIC\|ip-172-20-156-228.us-east-2.compute.internal#producer-2\| [thrd:app]: New local topic: 0022300646
%7\|1706584612.249\|TOPPARNEW\|ip-172-20-156-228.us-east-2.compute.internal#producer-2\| [thrd:app]: NEW 0022300646 [-1] 0x55f75a4d8240 refcnt 0x55f75a4d82d0 (at rd_kafka_topic_new0:472)
%7\|1706584613.068\|NOINFO\|ip-172-20-156-228.us-east-2.compute.internal#producer-2\| [thrd:main]: Topic 0022300646 metadata information unknown
%7\|1706584613.068\|NOINFO\|ip-172-20-156-228.us-east-2.compute.internal#producer-2\| [thrd:main]: Topic 0022300646 partition count is zero: should refresh metadata
%7\|1706584613.068\|CONNECT\|ip-172-20-156-228.us-east-2.compute.internal#producer-2\| [thrd:main]: Cluster connection already in progress: refresh unavailable topics
%7\|1706584613.068\|METADATA\|ip-172-20-156-228.us-east-2.compute.internal#producer-2\| [thrd:main]: Hinted cache of 1/1 topic(s) being queried
%7\|1706584613.068\|METADATA\|ip-172-20-156-228.us-east-2.compute.internal#producer-2\| [thrd:main]: Skipping metadata refresh of 1 topic(s): refresh unavailable topics: no usable brokers
%7\|1706584613.068\|CONNECT\|ip-172-20-156-228.us-east-2.compute.internal#producer-2\| [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 49ms: no cluster connection
%7\|1706584614.068\|NOINFO\|ip-172-20-156-228.us-east-2.compute.internal#producer-2\| [thrd:main]: Topic 0022300646 metadata information unknown
%7\|1706584614.068\|NOINFO\|ip-172-20-156-228.us-east-2.compute.internal#producer-2\| [thrd:main]: Topic 0022300646 partition count is zero: should refresh metadata
%7\|1706584614.068\|CONNECT\|ip-172-20-156-228.us-east-2.compute.internal#producer-2\| [thrd:main]: Cluster connection already in progress: refresh unavailable topics
%7\|1706584614.068\|METADATA\|ip-172-20-156-228.us-east-2.compute.internal#producer-2\| [thrd:main]: Hinted cache of 1/1 topic(s) being queried
%7\|1706584614.068\|METADATA\|ip-172-20-156-228.us-east-2.compute.internal#producer-2\| [thrd:main]: Skipping metadata refresh of 1 topic(s): refresh unavailable topics: no usable brokers
%7\|1706584614.068\|CONNECT\|ip-172-20-156-228.us-east-2.compute.internal#producer-2\| [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 49ms: no cluster connection
  • Critical issue

Thank you!

commented

I was using the config field for other kafka python library "sasl.oauthbearer.config" instead of using "oauth_cb".