Cannot install confluent-kafka with provided instructions on Apple's M1

Question

Cannot install confluent-kafka with provided instructions on Apple's M1

armenzg opened this issue 3 years ago · comments

Description

I cannot install confluent-kafka on Apple's M1 host with neither Python 3.8 nor Python 3.9 with the provided instructions.

Workaround

This is a combination of what @evgmoskalenko posted here and what @emrahgunduz posted here:

git clone https://github.com/edenhill/librdkafka.git
cd librdkafka
# I believe this step needs to happen before exporting the flags
brew install openssl zstd pkg-config
# For compilers to find openssl@1.1 you may need to set:
export CPPFLAGS="-I/opt/homebrew/opt/openssl@1.1/include"
export LDFLAGS="-L/opt/homebrew/opt/openssl@1.1/lib"
# From the librdkafka README, I believe we can skip this step since *just* configure is recommended
# ./configure --install-deps
./configure
make
sudo make install

Once the above steps were taken I was able to install the Python package confluent-kafka on Python 3.8.10.

❯ python -c "from confluent_kafka import version, libversion; print(version()); print(libversion())"
('1.7.0', 17235968)
('1.8.0-RC1-4-gc4d569', 17301759)

How to reproduce

brew upgrade
brew install librdkafka
# I've also done tried this with Python 3.8.10
pyenv install 3.9.6 && pyenv local 3.9.6
python -m venv venv && source venv/bin/activate && pip install -U wheel pip && pip install confluent-kafka

Versions:

❯ brew -v
Homebrew 3.2.9
Homebrew/homebrew-core (git revision 7fcca92399; last commit 2021-08-24)
Homebrew/homebrew-cask (git revision fc93850e89; last commit 2021-08-24)
❯ pyenv -v
pyenv 2.0.4
❯ brew info librdkafka
librdkafka: stable 1.7.0 (bottled), HEAD
Apache Kafka C/C++ library
https://github.com/edenhill/librdkafka
/opt/homebrew/Cellar/librdkafka/1.7.0 (36 files, 4.8MB) *
  Poured from bottle on 2021-08-24 at 10:22:09
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/librdkafka.rb
License: BSD-2-Clause
==> Dependencies
Build: pkg-config ✔, python@3.9 ✔
Required: lz4 ✔, lzlib ✔, openssl@1.1 ✔, zstd ✔
==> Options
--HEAD
	Install HEAD version
==> Analytics
install: 5,421 (30 days), 17,740 (90 days), 89,580 (365 days)
install-on-request: 3,440 (30 days), 11,254 (90 days), 55,667 (365 days)
build-error: 0 (30 days)

Checklist

Please provide the following information:

confluent-kafka-python and librdkafka version (confluent_kafka.version() and confluent_kafka.libversion()):
Apache Kafka broker version:
Client configuration: {...}
Operating system:
Provide client logs (with 'debug': '..' as necessary)
Provide broker log excerpts
Critical issue

Version():                                    ('1.7.0', 17235968)
libversion():                                ('1.8.0-RC1-4-gc4d569', 17301759)
Apache Kafka broker version:   Not applicable
Client configuration:                  Not applicable
OS:                                              Apple Big Sur 11.5.2 - Apple M1
Provide client logs:                    Not applicable
Provide broker log excerpts:     Not applicable
Critical issue:                             It is not because there's a work around

Log:

❯ python -m venv venv && source venv/bin/activate && pip install -U wheel pip && pip install confluent-kafka
Collecting wheel
  Using cached wheel-0.37.0-py2.py3-none-any.whl (35 kB)
Requirement already satisfied: pip in ./venv/lib/python3.9/site-packages (21.1.3)
Collecting pip
  Using cached pip-21.2.4-py3-none-any.whl (1.6 MB)
Installing collected packages: wheel, pip
  Attempting uninstall: pip
    Found existing installation: pip 21.1.3
    Uninstalling pip-21.1.3:
      Successfully uninstalled pip-21.1.3
Successfully installed pip-21.2.4 wheel-0.37.0
Collecting confluent-kafka
  Using cached confluent-kafka-1.7.0.tar.gz (103 kB)
Building wheels for collected packages: confluent-kafka
  Building wheel for confluent-kafka (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /Users/armenzg/code/M1/3_9/venv/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4/confluent-kafka_0f03988f28364fb2b8a67d82b84d8bdc/setup.py'"'"'; __file__='"'"'/private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4/confluent-kafka_0f03988f28364fb2b8a67d82b84d8bdc/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-wheel-xeoxwvu4
       cwd: /private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4/confluent-kafka_0f03988f28364fb2b8a67d82b84d8bdc/
  Complete output (55 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.macosx-11.5-arm64-3.9
  creating build/lib.macosx-11.5-arm64-3.9/confluent_kafka
  copying src/confluent_kafka/error.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka
  copying src/confluent_kafka/serializing_producer.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka
  copying src/confluent_kafka/__init__.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka
  copying src/confluent_kafka/deserializing_consumer.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka
  creating build/lib.macosx-11.5-arm64-3.9/confluent_kafka/schema_registry
  copying src/confluent_kafka/schema_registry/avro.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/schema_registry
  copying src/confluent_kafka/schema_registry/error.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/schema_registry
  copying src/confluent_kafka/schema_registry/__init__.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/schema_registry
  copying src/confluent_kafka/schema_registry/json_schema.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/schema_registry
  copying src/confluent_kafka/schema_registry/schema_registry_client.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/schema_registry
  copying src/confluent_kafka/schema_registry/protobuf.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/schema_registry
  creating build/lib.macosx-11.5-arm64-3.9/confluent_kafka/serialization
  copying src/confluent_kafka/serialization/__init__.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/serialization
  creating build/lib.macosx-11.5-arm64-3.9/confluent_kafka/admin
  copying src/confluent_kafka/admin/__init__.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/admin
  creating build/lib.macosx-11.5-arm64-3.9/confluent_kafka/avro
  copying src/confluent_kafka/avro/error.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/avro
  copying src/confluent_kafka/avro/__init__.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/avro
  copying src/confluent_kafka/avro/cached_schema_registry_client.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/avro
  copying src/confluent_kafka/avro/load.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/avro
  creating build/lib.macosx-11.5-arm64-3.9/confluent_kafka/kafkatest
  copying src/confluent_kafka/kafkatest/verifiable_client.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/kafkatest
  copying src/confluent_kafka/kafkatest/__init__.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/kafkatest
  copying src/confluent_kafka/kafkatest/verifiable_consumer.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/kafkatest
  copying src/confluent_kafka/kafkatest/verifiable_producer.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/kafkatest
  creating build/lib.macosx-11.5-arm64-3.9/confluent_kafka/avro/serializer
  copying src/confluent_kafka/avro/serializer/__init__.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/avro/serializer
  copying src/confluent_kafka/avro/serializer/message_serializer.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/avro/serializer
  running build_ext
  building 'confluent_kafka.cimpl' extension
  creating build/temp.macosx-11.5-arm64-3.9
  creating build/temp.macosx-11.5-arm64-3.9/private
  creating build/temp.macosx-11.5-arm64-3.9/private/var
  creating build/temp.macosx-11.5-arm64-3.9/private/var/folders
  creating build/temp.macosx-11.5-arm64-3.9/private/var/folders/j1
  creating build/temp.macosx-11.5-arm64-3.9/private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn
  creating build/temp.macosx-11.5-arm64-3.9/private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T
  creating build/temp.macosx-11.5-arm64-3.9/private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4
  creating build/temp.macosx-11.5-arm64-3.9/private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4/confluent-kafka_0f03988f28364fb2b8a67d82b84d8bdc
  creating build/temp.macosx-11.5-arm64-3.9/private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4/confluent-kafka_0f03988f28364fb2b8a67d82b84d8bdc/src
  creating build/temp.macosx-11.5-arm64-3.9/private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4/confluent-kafka_0f03988f28364fb2b8a67d82b84d8bdc/src/confluent_kafka
  creating build/temp.macosx-11.5-arm64-3.9/private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4/confluent-kafka_0f03988f28364fb2b8a67d82b84d8bdc/src/confluent_kafka/src
  clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Users/armenzg/code/M1/3_9/venv/include -I/Users/armenzg/.pyenv/versions/3.9.6/include/python3.9 -c /private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4/confluent-kafka_0f03988f28364fb2b8a67d82b84d8bdc/src/confluent_kafka/src/Admin.c -o build/temp.macosx-11.5-arm64-3.9/private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4/confluent-kafka_0f03988f28364fb2b8a67d82b84d8bdc/src/confluent_kafka/src/Admin.o
  In file included from /private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4/confluent-kafka_0f03988f28364fb2b8a67d82b84d8bdc/src/confluent_kafka/src/Admin.c:17:
  /private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4/confluent-kafka_0f03988f28364fb2b8a67d82b84d8bdc/src/confluent_kafka/src/confluent_kafka.h:23:10: fatal error: 'librdkafka/rdkafka.h' file not found
  #include <librdkafka/rdkafka.h>
           ^~~~~~~~~~~~~~~~~~~~~~
  1 error generated.
  error: command '/usr/bin/clang' failed with exit code 1
  ----------------------------------------
  ERROR: Failed building wheel for confluent-kafka
  Running setup.py clean for confluent-kafka
Failed to build confluent-kafka
Installing collected packages: confluent-kafka
    Running setup.py install for confluent-kafka ... error
    ERROR: Command errored out with exit status 1:
     command: /Users/armenzg/code/M1/3_9/venv/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4/confluent-kafka_0f03988f28364fb2b8a67d82b84d8bdc/setup.py'"'"'; __file__='"'"'/private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4/confluent-kafka_0f03988f28364fb2b8a67d82b84d8bdc/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-record-jt0ajm2s/install-record.txt --single-version-externally-managed --compile --install-headers /Users/armenzg/code/M1/3_9/venv/include/site/python3.9/confluent-kafka
         cwd: /private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4/confluent-kafka_0f03988f28364fb2b8a67d82b84d8bdc/
    Complete output (55 lines):
    running install
    running build
    running build_py
    creating build
    creating build/lib.macosx-11.5-arm64-3.9
    creating build/lib.macosx-11.5-arm64-3.9/confluent_kafka
    copying src/confluent_kafka/error.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka
    copying src/confluent_kafka/serializing_producer.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka
    copying src/confluent_kafka/__init__.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka
    copying src/confluent_kafka/deserializing_consumer.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka
    creating build/lib.macosx-11.5-arm64-3.9/confluent_kafka/schema_registry
    copying src/confluent_kafka/schema_registry/avro.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/schema_registry
    copying src/confluent_kafka/schema_registry/error.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/schema_registry
    copying src/confluent_kafka/schema_registry/__init__.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/schema_registry
    copying src/confluent_kafka/schema_registry/json_schema.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/schema_registry
    copying src/confluent_kafka/schema_registry/schema_registry_client.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/schema_registry
    copying src/confluent_kafka/schema_registry/protobuf.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/schema_registry
    creating build/lib.macosx-11.5-arm64-3.9/confluent_kafka/serialization
    copying src/confluent_kafka/serialization/__init__.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/serialization
    creating build/lib.macosx-11.5-arm64-3.9/confluent_kafka/admin
    copying src/confluent_kafka/admin/__init__.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/admin
    creating build/lib.macosx-11.5-arm64-3.9/confluent_kafka/avro
    copying src/confluent_kafka/avro/error.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/avro
    copying src/confluent_kafka/avro/__init__.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/avro
    copying src/confluent_kafka/avro/cached_schema_registry_client.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/avro
    copying src/confluent_kafka/avro/load.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/avro
    creating build/lib.macosx-11.5-arm64-3.9/confluent_kafka/kafkatest
    copying src/confluent_kafka/kafkatest/verifiable_client.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/kafkatest
    copying src/confluent_kafka/kafkatest/__init__.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/kafkatest
    copying src/confluent_kafka/kafkatest/verifiable_consumer.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/kafkatest
    copying src/confluent_kafka/kafkatest/verifiable_producer.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/kafkatest
    creating build/lib.macosx-11.5-arm64-3.9/confluent_kafka/avro/serializer
    copying src/confluent_kafka/avro/serializer/__init__.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/avro/serializer
    copying src/confluent_kafka/avro/serializer/message_serializer.py -> build/lib.macosx-11.5-arm64-3.9/confluent_kafka/avro/serializer
    running build_ext
    building 'confluent_kafka.cimpl' extension
    creating build/temp.macosx-11.5-arm64-3.9
    creating build/temp.macosx-11.5-arm64-3.9/private
    creating build/temp.macosx-11.5-arm64-3.9/private/var
    creating build/temp.macosx-11.5-arm64-3.9/private/var/folders
    creating build/temp.macosx-11.5-arm64-3.9/private/var/folders/j1
    creating build/temp.macosx-11.5-arm64-3.9/private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn
    creating build/temp.macosx-11.5-arm64-3.9/private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T
    creating build/temp.macosx-11.5-arm64-3.9/private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4
    creating build/temp.macosx-11.5-arm64-3.9/private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4/confluent-kafka_0f03988f28364fb2b8a67d82b84d8bdc
    creating build/temp.macosx-11.5-arm64-3.9/private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4/confluent-kafka_0f03988f28364fb2b8a67d82b84d8bdc/src
    creating build/temp.macosx-11.5-arm64-3.9/private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4/confluent-kafka_0f03988f28364fb2b8a67d82b84d8bdc/src/confluent_kafka
    creating build/temp.macosx-11.5-arm64-3.9/private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4/confluent-kafka_0f03988f28364fb2b8a67d82b84d8bdc/src/confluent_kafka/src
    clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Users/armenzg/code/M1/3_9/venv/include -I/Users/armenzg/.pyenv/versions/3.9.6/include/python3.9 -c /private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4/confluent-kafka_0f03988f28364fb2b8a67d82b84d8bdc/src/confluent_kafka/src/Admin.c -o build/temp.macosx-11.5-arm64-3.9/private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4/confluent-kafka_0f03988f28364fb2b8a67d82b84d8bdc/src/confluent_kafka/src/Admin.o
    In file included from /private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4/confluent-kafka_0f03988f28364fb2b8a67d82b84d8bdc/src/confluent_kafka/src/Admin.c:17:
    /private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4/confluent-kafka_0f03988f28364fb2b8a67d82b84d8bdc/src/confluent_kafka/src/confluent_kafka.h:23:10: fatal error: 'librdkafka/rdkafka.h' file not found
    #include <librdkafka/rdkafka.h>
             ^~~~~~~~~~~~~~~~~~~~~~
    1 error generated.
    error: command '/usr/bin/clang' failed with exit code 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /Users/armenzg/code/M1/3_9/venv/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4/confluent-kafka_0f03988f28364fb2b8a67d82b84d8bdc/setup.py'"'"'; __file__='"'"'/private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-64j5n_j4/confluent-kafka_0f03988f28364fb2b8a67d82b84d8bdc/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-record-jt0ajm2s/install-record.txt --single-version-externally-managed --compile --install-headers /Users/armenzg/code/M1/3_9/venv/include/site/python3.9/confluent-kafka Check the logs for full command output.

Armen Zambrano G. · Answer 1 · Tue Sep 07 2021 21:12:07 GMT+0800 (China Standard Time)

@jliunyu Hi! Would you mind providing some guidance on how to fix this? Or point me to someone that could give some pointers? Thanks!

Jing Liu · Answer 2 · Wed Sep 08 2021 06:00:55 GMT+0800 (China Standard Time)

@armenzg, I'm sorry for the late response. We didn't have a chance to test on Apple's M1 host yet.

@edenhill, do we have any plan for testing on Apple's M1 host?

Jing Liu · Answer 3 · Thu Sep 09 2021 02:21:22 GMT+0800 (China Standard Time)

Hi @armenzg , this looks like a path issue. Because librdkafka is installed with brew, but python doesn't look into the brew directories. So our suggestion is using the workaround you already used or find a way that python can find the brew installed packages.
From our side, we are looking into providing M1 binary wheels.

Armen Zambrano G. · Answer 4 · Thu Sep 16 2021 21:33:02 GMT+0800 (China Standard Time)

Hi @jliunyu my apologies on the delay. I have no idea how I missed your reply.

I did some work on producing arm64 wheels in here. Cross-compiling does not work.

Armen Zambrano G. · Answer 5 · Fri Sep 17 2021 04:13:13 GMT+0800 (China Standard Time)

Hi @jliunyu my apologies on the delay. I have no idea how I missed your reply.

I did some work on producing arm64 wheels in here. Cross-compiling does not work. You need an arm64 machine to produce it.

Armen Zambrano G. · Answer 6 · Fri Nov 19 2021 21:38:27 GMT+0800 (China Standard Time)

@jliunyu I figured this out.

On Apple M1 hosts, all its needed is this:

CPATH=/opt/homebrew/Cellar/librdkafka/1.8.2/include pip install confluent-kafka

Jing Liu · Answer 7 · Mon Nov 22 2021 06:41:40 GMT+0800 (China Standard Time)

@jliunyu I figured this out.

On Apple M1 hosts, all its needed is this:

CPATH=/opt/homebrew/Cellar/librdkafka/1.8.2/include pip install confluent-kafka

Thanks for your update @armenzg .

pthalasta · Answer 8 · Fri Jan 14 2022 04:46:00 GMT+0800 (China Standard Time)

Is this issue fixed? I'm seeing issues using confluent-kafka lib on M1

from confluent_kafka.cimpl import Consumer as _ConsumerImpl
ImportError: dlopen(/opt/homebrew/lib/python3.9/site-packages/confluent_kafka/cimpl.cpython-39-darwin.so, 0x0002): tried: '/opt/homebrew/lib/python3.9/site-packages/confluent_kafka/cimpl.cpython-39-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')), '/usr/local/lib/cimpl.cpython-39-darwin.so' (no such file), '/usr/lib/cimpl.cpython-39-darwin.so' (no such file)

Jing Liu · Answer 9 · Thu Jan 20 2022 09:58:47 GMT+0800 (China Standard Time)

@edenhill is working on this, it will be more compatible with M1

Marc · Answer 10 · Thu Jan 20 2022 19:39:16 GMT+0800 (China Standard Time)

CPATH was not quite working for me, but this did:

C_INCLUDE_PATH=/opt/homebrew/Cellar/librdkafka/1.8.2/include LIBRARY_PATH=/opt/homebrew/Cellar/librdkafka/1.8.2/lib pip install confluent_kafka

Remember to change the version to the librdkafka version you have installed through brew.

Yuto Takano · Answer 11 · Wed Jan 26 2022 23:14:30 GMT+0800 (China Standard Time)

CPATH was not quite working for me, but this did:

C_INCLUDE_PATH=/opt/homebrew/Cellar/librdkafka/1.8.2/include LIBRARY_PATH=/opt/homebrew/Cellar/librdkafka/1.8.2/lib pip install confluent_kafka

Remember to change the version to the librdkafka version you have installed through brew.

This works. Unfortunately, I can't seem to get it to work in a Python Docker image, something like

FROM python:3.9
WORKDIR /code
COPY requirements.txt requirements.txt
RUN pip install -r requirements.txt
...

fails, as I cannot pass those environment variables to the pip install command. I also do not want to change the Dockerfile since this is shared with the entire team, not just me.

pthalasta · Answer 12 · Thu Jan 27 2022 03:05:49 GMT+0800 (China Standard Time)

CPATH was not quite working for me, but this did:

C_INCLUDE_PATH=/opt/homebrew/Cellar/librdkafka/1.8.2/include LIBRARY_PATH=/opt/homebrew/Cellar/librdkafka/1.8.2/lib pip install confluent_kafka

Remember to change the version to the librdkafka version you have installed through brew.

This did not work on my MAC. I still see the same error

from confluent_kafka.cimpl import Consumer as _ConsumerImpl
ImportError: dlopen(/opt/homebrew/lib/python3.9/site-packages/confluent_kafka/cimpl.cpython-39-darwin.so, 0x0002): tried: '/opt/homebrew/lib/python3.9/site-packages/confluent_kafka/cimpl.cpython-39-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')), '/usr/local/lib/cimpl.cpython-39-darwin.so' (no such file), '/usr/lib/cimpl.cpython-39-darwin.so' (no such file)

Jing Liu · Answer 13 · Sat Jan 29 2022 09:47:42 GMT+0800 (China Standard Time)

Link the same issue with discussion to this one: #1025

Khachik Smbatyan · Answer 14 · Tue Feb 08 2022 19:39:50 GMT+0800 (China Standard Time)

Hi @yutotakano 👋
I have the same issue and changing the way of building the image helped me to solve it.
In case you still experience the issue,
you can solve it by setting the platform to linux/amd64.
docker build --platform linux/amd64 --tag some-tag .

Hope this will help 🙂

Useful links

https://www.docker.com/blog/faster-multi-platform-builds-dockerfile-cross-compilation-guide/

ygag-sabarinath · Answer 15 · Fri Mar 11 2022 21:29:36 GMT+0800 (China Standard Time)

@pthalasta thanks

Richard Lee · Answer 16 · Sat Apr 30 2022 05:42:24 GMT+0800 (China Standard Time)

also not working for me. I have set CPATH, C_INCLUDE_PATH, and LIBRARY_PATH, but it still doesn't find the brew installed librdkafka headers on an M1 mac.

      clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -I/Users/rdlee/Documents/tivo-research/git/kafka-copier/mirrormaker2-config-manager/venv/include -I/opt/homebrew/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c /private/var/folders/3j/k8b_rph92hlf_0vx00nz4x_r0000gq/T/pip-install-64g22oz_/confluent-kafka_471c40aea8c44b25ac7262a4ce18c69b/src/confluent_kafka/src/Admin.c -o build/temp.macosx-12-arm64-3.9/private/var/folders/3j/k8b_rph92hlf_0vx00nz4x_r0000gq/T/pip-install-64g22oz_/confluent-kafka_471c40aea8c44b25ac7262a4ce18c69b/src/confluent_kafka/src/Admin.o
      In file included from /private/var/folders/3j/k8b_rph92hlf_0vx00nz4x_r0000gq/T/pip-install-64g22oz_/confluent-kafka_471c40aea8c44b25ac7262a4ce18c69b/src/confluent_kafka/src/Admin.c:17:
      /private/var/folders/3j/k8b_rph92hlf_0vx00nz4x_r0000gq/T/pip-install-64g22oz_/confluent-kafka_471c40aea8c44b25ac7262a4ce18c69b/src/confluent_kafka/src/confluent_kafka.h:23:10: fatal error: 'librdkafka/rdkafka.h' file not found
      #include <librdkafka/rdkafka.h>
               ^~~~~~~~~~~~~~~~~~~~~~
      1 error generated.
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]

yet

% ls -l $CPATH/librdkafka/rdkafka.h
-rwxr-xr-x  1 rdlee  admin  281064 Oct 11  2021 /opt/homebrew/include/librdkafka/rdkafka.h

Callam · Answer 17 · Mon May 16 2022 18:34:23 GMT+0800 (China Standard Time)

I had to set this environment variable to get this to work
C_INCLUDE_PATH=/opt/homebrew/Cellar/librdkafka/1.8.2/include LIBRARY_PATH=/opt/homebrew/Cellar/librdkafka/1.8.2/lib pip install confluent_kafka

Then I was able to successfully run pip install confluent-kafka

Tripti Mishra · Answer 18 · Tue May 17 2022 15:14:42 GMT+0800 (China Standard Time)

I am getting the following error while i try to install pip install confluent-kafka
error: subprocess-exited-with-error

× Running setup.py install for confluent_kafka did not run successfully.
│ exit code: 1
╰─> [55 lines of output]
running install
running build
running build_py
creating build
creating build/lib.macosx-10.9-universal2-3.10
creating build/lib.macosx-10.9-universal2-3.10/confluent_kafka
copying src/confluent_kafka/error.py -> build/lib.macosx-10.9-universal2-3.10/confluent_kafka
copying src/confluent_kafka/serializing_producer.py -> build/lib.macosx-10.9-universal2-3.10/confluent_kafka
copying src/confluent_kafka/init.py -> build/lib.macosx-10.9-universal2-3.10/confluent_kafka
copying src/confluent_kafka/deserializing_consumer.py -> build/lib.macosx-10.9-universal2-3.10/confluent_kafka
creating build/lib.macosx-10.9-universal2-3.10/confluent_kafka/schema_registry
copying src/confluent_kafka/schema_registry/avro.py -> build/lib.macosx-10.9-universal2-3.10/confluent_kafka/schema_registry
copying src/confluent_kafka/schema_registry/error.py -> build/lib.macosx-10.9-universal2-3.10/confluent_kafka/schema_registry
copying src/confluent_kafka/schema_registry/init.py -> build/lib.macosx-10.9-universal2-3.10/confluent_kafka/schema_registry
copying src/confluent_kafka/schema_registry/json_schema.py -> build/lib.macosx-10.9-universal2-3.10/confluent_kafka/schema_registry
copying src/confluent_kafka/schema_registry/schema_registry_client.py -> build/lib.macosx-10.9-universal2-3.10/confluent_kafka/schema_registry
copying src/confluent_kafka/schema_registry/protobuf.py -> build/lib.macosx-10.9-universal2-3.10/confluent_kafka/schema_registry
creating build/lib.macosx-10.9-universal2-3.10/confluent_kafka/serialization
copying src/confluent_kafka/serialization/init.py -> build/lib.macosx-10.9-universal2-3.10/confluent_kafka/serialization
creating build/lib.macosx-10.9-universal2-3.10/confluent_kafka/admin
copying src/confluent_kafka/admin/init.py -> build/lib.macosx-10.9-universal2-3.10/confluent_kafka/admin
creating build/lib.macosx-10.9-universal2-3.10/confluent_kafka/avro
copying src/confluent_kafka/avro/error.py -> build/lib.macosx-10.9-universal2-3.10/confluent_kafka/avro
copying src/confluent_kafka/avro/init.py -> build/lib.macosx-10.9-universal2-3.10/confluent_kafka/avro
copying src/confluent_kafka/avro/cached_schema_registry_client.py -> build/lib.macosx-10.9-universal2-3.10/confluent_kafka/avro
copying src/confluent_kafka/avro/load.py -> build/lib.macosx-10.9-universal2-3.10/confluent_kafka/avro
creating build/lib.macosx-10.9-universal2-3.10/confluent_kafka/kafkatest
copying src/confluent_kafka/kafkatest/verifiable_client.py -> build/lib.macosx-10.9-universal2-3.10/confluent_kafka/kafkatest
copying src/confluent_kafka/kafkatest/init.py -> build/lib.macosx-10.9-universal2-3.10/confluent_kafka/kafkatest
copying src/confluent_kafka/kafkatest/verifiable_consumer.py -> build/lib.macosx-10.9-universal2-3.10/confluent_kafka/kafkatest
copying src/confluent_kafka/kafkatest/verifiable_producer.py -> build/lib.macosx-10.9-universal2-3.10/confluent_kafka/kafkatest
creating build/lib.macosx-10.9-universal2-3.10/confluent_kafka/avro/serializer
copying src/confluent_kafka/avro/serializer/init.py -> build/lib.macosx-10.9-universal2-3.10/confluent_kafka/avro/serializer
copying src/confluent_kafka/avro/serializer/message_serializer.py -> build/lib.macosx-10.9-universal2-3.10/confluent_kafka/avro/serializer
running build_ext
building 'confluent_kafka.cimpl' extension
creating build/temp.macosx-10.9-universal2-3.10
creating build/temp.macosx-10.9-universal2-3.10/private
creating build/temp.macosx-10.9-universal2-3.10/private/var
creating build/temp.macosx-10.9-universal2-3.10/private/var/folders
creating build/temp.macosx-10.9-universal2-3.10/private/var/folders/2k
creating build/temp.macosx-10.9-universal2-3.10/private/var/folders/2k/t0j9lrz10_l8fl4bsld9_01m0000gn
creating build/temp.macosx-10.9-universal2-3.10/private/var/folders/2k/t0j9lrz10_l8fl4bsld9_01m0000gn/T
creating build/temp.macosx-10.9-universal2-3.10/private/var/folders/2k/t0j9lrz10_l8fl4bsld9_01m0000gn/T/pip-install-e8te3y40
creating build/temp.macosx-10.9-universal2-3.10/private/var/folders/2k/t0j9lrz10_l8fl4bsld9_01m0000gn/T/pip-install-e8te3y40/confluent-kafka_b23fb0e1e0df4c6492d3b21342b4e7af
creating build/temp.macosx-10.9-universal2-3.10/private/var/folders/2k/t0j9lrz10_l8fl4bsld9_01m0000gn/T/pip-install-e8te3y40/confluent-kafka_b23fb0e1e0df4c6492d3b21342b4e7af/src
creating build/temp.macosx-10.9-universal2-3.10/private/var/folders/2k/t0j9lrz10_l8fl4bsld9_01m0000gn/T/pip-install-e8te3y40/confluent-kafka_b23fb0e1e0df4c6492d3b21342b4e7af/src/confluent_kafka
creating build/temp.macosx-10.9-universal2-3.10/private/var/folders/2k/t0j9lrz10_l8fl4bsld9_01m0000gn/T/pip-install-e8te3y40/confluent-kafka_b23fb0e1e0df4c6492d3b21342b4e7af/src/confluent_kafka/src
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch arm64 -arch x86_64 -g -I/Library/Frameworks/Python.framework/Versions/3.10/include/python3.10 -c /private/var/folders/2k/t0j9lrz10_l8fl4bsld9_01m0000gn/T/pip-install-e8te3y40/confluent-kafka_b23fb0e1e0df4c6492d3b21342b4e7af/src/confluent_kafka/src/Admin.c -o build/temp.macosx-10.9-universal2-3.10/private/var/folders/2k/t0j9lrz10_l8fl4bsld9_01m0000gn/T/pip-install-e8te3y40/confluent-kafka_b23fb0e1e0df4c6492d3b21342b4e7af/src/confluent_kafka/src/Admin.o
In file included from /private/var/folders/2k/t0j9lrz10_l8fl4bsld9_01m0000gn/T/pip-install-e8te3y40/confluent-kafka_b23fb0e1e0df4c6492d3b21342b4e7af/src/confluent_kafka/src/Admin.c:17:
/private/var/folders/2k/t0j9lrz10_l8fl4bsld9_01m0000gn/T/pip-install-e8te3y40/confluent-kafka_b23fb0e1e0df4c6492d3b21342b4e7af/src/confluent_kafka/src/confluent_kafka.h:23:10: fatal error: 'librdkafka/rdkafka.h' file not found
#include <librdkafka/rdkafka.h>
^~~~~~~~~~~~~~~~~~~~~~
1 error generated.
error: command '/usr/bin/clang' failed with exit code 1
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> confluent_kafka

Please help

Ramon Niebla · Answer 19 · Thu May 19 2022 13:18:18 GMT+0800 (China Standard Time)

I'm also getting the same error above when pip installing in a dockerfile!

Works fine when emulating per: #1190 (comment)

Hyun-Jin Jeong · Answer 20 · Wed May 25 2022 23:37:41 GMT+0800 (China Standard Time)

If you put the lines in the shell script file such as .zshrc and .bashrc, you should use them with export:

export C_INCLUDE_PATH=/opt/homebrew/Cellar/librdkafka/1.8.2/include
export LIBRARY_PATH=/opt/homebrew/Cellar/librdkafka/1.8.2/lib

Nikolay Markov · Answer 21 · Sat Jun 11 2022 15:26:35 GMT+0800 (China Standard Time)

It's works for my M1 !
Thanks

Carlos Tangerino · Answer 22 · Wed Jun 22 2022 13:34:01 GMT+0800 (China Standard Time)

Even with the suggestions above, I get

❯ brew reinstall librdkafka
==> Downloading https://ghcr.io/v2/homebrew/core/librdkafka/manifests/1.9.0
Already downloaded: /Users/carlostangerino/Library/Caches/Homebrew/downloads/d7e9a712f8f7e97e8509a83f497d9e884db3bc48c7f98be129f912ce3be53901--librdkafka-1.9.0.bottle_manifest.json
==> Downloading https://ghcr.io/v2/homebrew/core/librdkafka/blobs/sha256:bbe7efc09d58c2fd56280eb749883b330d7f066735be640911cea5e008bab6fa
Already downloaded: /Users/carlostangerino/Library/Caches/Homebrew/downloads/8353d49b621f5600c8f626c36681967098a4120527697f559c25b796e909a394--librdkafka--1.9.0.arm64_monterey.bottle.tar.gz
==> Reinstalling librdkafka
==> Pouring librdkafka--1.9.0.arm64_monterey.bottle.tar.gz
🍺  /opt/homebrew/Cellar/librdkafka/1.9.0: 38 files, 7.6MB
==> Running `brew cleanup librdkafka`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).

C_INCLUDE_PATH=/opt/homebrew/Cellar/librdkafka/1.9.0/include 
LIBRARY_PATH=/opt/homebrew/Cellar/librdkafka/1.9.0/lib

❯ pip3 install confluent-kafka
DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
Collecting confluent-kafka
  Using cached confluent-kafka-1.9.0.tar.gz (110 kB)
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: confluent-kafka
  Building wheel for confluent-kafka (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [117 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.macosx-12-arm64-cpython-39
      creating build/lib.macosx-12-arm64-cpython-39/confluent_kafka
      copying src/confluent_kafka/error.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka
      copying src/confluent_kafka/serializing_producer.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka
      copying src/confluent_kafka/__init__.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka
      copying src/confluent_kafka/deserializing_consumer.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka
      creating build/lib.macosx-12-arm64-cpython-39/confluent_kafka/schema_registry
      copying src/confluent_kafka/schema_registry/avro.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/schema_registry
      copying src/confluent_kafka/schema_registry/error.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/schema_registry
      copying src/confluent_kafka/schema_registry/__init__.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/schema_registry
      copying src/confluent_kafka/schema_registry/json_schema.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/schema_registry
      copying src/confluent_kafka/schema_registry/schema_registry_client.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/schema_registry
      copying src/confluent_kafka/schema_registry/protobuf.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/schema_registry
      creating build/lib.macosx-12-arm64-cpython-39/confluent_kafka/serialization
      copying src/confluent_kafka/serialization/__init__.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/serialization
      creating build/lib.macosx-12-arm64-cpython-39/confluent_kafka/admin
      copying src/confluent_kafka/admin/_acl.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/admin
      copying src/confluent_kafka/admin/_resource.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/admin
      copying src/confluent_kafka/admin/__init__.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/admin
      copying src/confluent_kafka/admin/_config.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/admin
      creating build/lib.macosx-12-arm64-cpython-39/confluent_kafka/avro
      copying src/confluent_kafka/avro/error.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/avro
      copying src/confluent_kafka/avro/__init__.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/avro
      copying src/confluent_kafka/avro/cached_schema_registry_client.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/avro
      copying src/confluent_kafka/avro/load.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/avro
      creating build/lib.macosx-12-arm64-cpython-39/confluent_kafka/kafkatest
      copying src/confluent_kafka/kafkatest/verifiable_client.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/kafkatest
      copying src/confluent_kafka/kafkatest/__init__.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/kafkatest
      copying src/confluent_kafka/kafkatest/verifiable_consumer.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/kafkatest
      copying src/confluent_kafka/kafkatest/verifiable_producer.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/kafkatest
      creating build/lib.macosx-12-arm64-cpython-39/confluent_kafka/avro/serializer
      copying src/confluent_kafka/avro/serializer/__init__.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/avro/serializer
      copying src/confluent_kafka/avro/serializer/message_serializer.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/avro/serializer
      running build_ext
      building 'confluent_kafka.cimpl' extension
      creating build/temp.macosx-12-arm64-cpython-39
      creating build/temp.macosx-12-arm64-cpython-39/private
      creating build/temp.macosx-12-arm64-cpython-39/private/var
      creating build/temp.macosx-12-arm64-cpython-39/private/var/folders
      creating build/temp.macosx-12-arm64-cpython-39/private/var/folders/wr
      creating build/temp.macosx-12-arm64-cpython-39/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn
      creating build/temp.macosx-12-arm64-cpython-39/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T
      creating build/temp.macosx-12-arm64-cpython-39/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3
      creating build/temp.macosx-12-arm64-cpython-39/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98
      creating build/temp.macosx-12-arm64-cpython-39/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src
      creating build/temp.macosx-12-arm64-cpython-39/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka
      creating build/temp.macosx-12-arm64-cpython-39/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src
      clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/opt/homebrew/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c /private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c -o build/temp.macosx-12-arm64-cpython-39/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.o
      In file included from /private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:17:
      /private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/confluent_kafka.h:64:2: error: "confluent-kafka-python requires librdkafka v1.9.0 or later. Install the latest version of librdkafka from Homebrew by running `brew install librdkafka` or `brew upgrade librdkafka`"
      #error "confluent-kafka-python requires librdkafka v1.9.0 or later. Install the latest version of librdkafka from Homebrew by running `brew install librdkafka` or `brew upgrade librdkafka`"
       ^
      /private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:156:8: error: unknown type name 'rd_kafka_AclBinding_t'
      static rd_kafka_AclBinding_t *
             ^
      /private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:162:9: error: use of undeclared identifier 'rd_kafka_AclBinding_t'
              rd_kafka_AclBinding_t *ret = NULL;
              ^
      /private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:162:32: error: use of undeclared identifier 'ret'
              rd_kafka_AclBinding_t *ret = NULL;
                                     ^
      /private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:172:21: error: use of undeclared identifier 'ret'
                          ret = rd_kafka_AclBinding_new(restype, resname, \
                          ^
      /private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:172:27: error: implicit declaration of function 'rd_kafka_AclBinding_new' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
                          ret = rd_kafka_AclBinding_new(restype, resname, \
                                ^
      /private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:179:16: error: use of undeclared identifier 'ret'
              return ret;
                     ^
      /private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:185:8: error: unknown type name 'rd_kafka_AclBindingFilter_t'
      static rd_kafka_AclBindingFilter_t*
             ^
      /private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:192:9: error: use of undeclared identifier 'rd_kafka_AclBindingFilter_t'
              rd_kafka_AclBindingFilter_t* ret = NULL;
              ^
      /private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:192:38: error: use of undeclared identifier 'ret'
              rd_kafka_AclBindingFilter_t* ret = NULL;
                                           ^
      /private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:201:21: error: use of undeclared identifier 'ret'
                          ret = rd_kafka_AclBindingFilter_new(restype, resname, \
                          ^
      /private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:201:27: error: implicit declaration of function 'rd_kafka_AclBindingFilter_new' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
                          ret = rd_kafka_AclBindingFilter_new(restype, resname, \
                                ^
      /private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:208:16: error: use of undeclared identifier 'ret'
              return ret;
                     ^
      /private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:1045:9: error: use of undeclared identifier 'rd_kafka_AclBinding_t'
              rd_kafka_AclBinding_t **c_objs = NULL;
              ^
      /private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:1045:33: error: use of undeclared identifier 'c_objs'
              rd_kafka_AclBinding_t **c_objs = NULL;
                                      ^
      /private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:1081:46: error: use of undeclared identifier 'RD_KAFKA_ADMIN_OP_CREATEACLS'; did you mean 'RD_KAFKA_ADMIN_OP_CREATETOPICS'?
              c_options = Admin_options_to_c(self, RD_KAFKA_ADMIN_OP_CREATEACLS,
                                                   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
                                                   RD_KAFKA_ADMIN_OP_CREATETOPICS
      /usr/local/include/librdkafka/rdkafka.h:6117:9: note: 'RD_KAFKA_ADMIN_OP_CREATETOPICS' declared here
              RD_KAFKA_ADMIN_OP_CREATETOPICS,     /**< CreateTopics */
              ^
      /private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:1095:9: error: use of undeclared identifier 'c_objs'
              c_objs = malloc(sizeof(*c_objs) * cnt);
              ^
      /private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:1095:33: error: use of undeclared identifier 'c_objs'
              c_objs = malloc(sizeof(*c_objs) * cnt);
                                      ^
      /private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:1112:17: error: use of undeclared identifier 'c_objs'
                      c_objs[i] = Admin_py_to_c_AclBinding(res, errstr, sizeof(errstr));
                      ^
      fatal error: too many errors emitted, stopping now [-ferror-limit=]
      20 errors generated.
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for confluent-kafka
  Running setup.py clean for confluent-kafka
Failed to build confluent-kafka
Installing collected packages: confluent-kafka
  DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
  Running setup.py install for confluent-kafka ... error
  error: subprocess-exited-with-error

  × Running setup.py install for confluent-kafka did not run successfully.
  │ exit code: 1
  ╰─> [119 lines of output]
      running install
      /opt/homebrew/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_py
      creating build
      creating build/lib.macosx-12-arm64-cpython-39
      creating build/lib.macosx-12-arm64-cpython-39/confluent_kafka
      copying src/confluent_kafka/error.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka
      copying src/confluent_kafka/serializing_producer.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka
      copying src/confluent_kafka/__init__.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka
      copying src/confluent_kafka/deserializing_consumer.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka
      creating build/lib.macosx-12-arm64-cpython-39/confluent_kafka/schema_registry
      copying src/confluent_kafka/schema_registry/avro.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/schema_registry
      copying src/confluent_kafka/schema_registry/error.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/schema_registry
      copying src/confluent_kafka/schema_registry/__init__.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/schema_registry
      copying src/confluent_kafka/schema_registry/json_schema.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/schema_registry
      copying src/confluent_kafka/schema_registry/schema_registry_client.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/schema_registry
      copying src/confluent_kafka/schema_registry/protobuf.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/schema_registry
      creating build/lib.macosx-12-arm64-cpython-39/confluent_kafka/serialization
      copying src/confluent_kafka/serialization/__init__.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/serialization
      creating build/lib.macosx-12-arm64-cpython-39/confluent_kafka/admin
      copying src/confluent_kafka/admin/_acl.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/admin
      copying src/confluent_kafka/admin/_resource.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/admin
      copying src/confluent_kafka/admin/__init__.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/admin
      copying src/confluent_kafka/admin/_config.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/admin
      creating build/lib.macosx-12-arm64-cpython-39/confluent_kafka/avro
      copying src/confluent_kafka/avro/error.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/avro
      copying src/confluent_kafka/avro/__init__.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/avro
      copying src/confluent_kafka/avro/cached_schema_registry_client.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/avro
      copying src/confluent_kafka/avro/load.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/avro
      creating build/lib.macosx-12-arm64-cpython-39/confluent_kafka/kafkatest
      copying src/confluent_kafka/kafkatest/verifiable_client.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/kafkatest
      copying src/confluent_kafka/kafkatest/__init__.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/kafkatest
      copying src/confluent_kafka/kafkatest/verifiable_consumer.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/kafkatest
      copying src/confluent_kafka/kafkatest/verifiable_producer.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/kafkatest
      creating build/lib.macosx-12-arm64-cpython-39/confluent_kafka/avro/serializer
      copying src/confluent_kafka/avro/serializer/__init__.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/avro/serializer
      copying src/confluent_kafka/avro/serializer/message_serializer.py -> build/lib.macosx-12-arm64-cpython-39/confluent_kafka/avro/serializer
      running build_ext
      building 'confluent_kafka.cimpl' extension
      creating build/temp.macosx-12-arm64-cpython-39
      creating build/temp.macosx-12-arm64-cpython-39/private
      creating build/temp.macosx-12-arm64-cpython-39/private/var
      creating build/temp.macosx-12-arm64-cpython-39/private/var/folders
      creating build/temp.macosx-12-arm64-cpython-39/private/var/folders/wr
      creating build/temp.macosx-12-arm64-cpython-39/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn
      creating build/temp.macosx-12-arm64-cpython-39/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T
      creating build/temp.macosx-12-arm64-cpython-39/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3
      creating build/temp.macosx-12-arm64-cpython-39/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98
      creating build/temp.macosx-12-arm64-cpython-39/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src
      creating build/temp.macosx-12-arm64-cpython-39/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka
      creating build/temp.macosx-12-arm64-cpython-39/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src
      clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/opt/homebrew/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c /private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c -o build/temp.macosx-12-arm64-cpython-39/private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.o
      In file included from /private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:17:
      /private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/confluent_kafka.h:64:2: error: "confluent-kafka-python requires librdkafka v1.9.0 or later. Install the latest version of librdkafka from Homebrew by running `brew install librdkafka` or `brew upgrade librdkafka`"
      #error "confluent-kafka-python requires librdkafka v1.9.0 or later. Install the latest version of librdkafka from Homebrew by running `brew install librdkafka` or `brew upgrade librdkafka`"
       ^
      /private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:156:8: error: unknown type name 'rd_kafka_AclBinding_t'
      static rd_kafka_AclBinding_t *
             ^
      /private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:162:9: error: use of undeclared identifier 'rd_kafka_AclBinding_t'
              rd_kafka_AclBinding_t *ret = NULL;
              ^
      /private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:162:32: error: use of undeclared identifier 'ret'
              rd_kafka_AclBinding_t *ret = NULL;
                                     ^
      /private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:172:21: error: use of undeclared identifier 'ret'
                          ret = rd_kafka_AclBinding_new(restype, resname, \
                          ^
      /private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:172:27: error: implicit declaration of function 'rd_kafka_AclBinding_new' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
                          ret = rd_kafka_AclBinding_new(restype, resname, \
                                ^
      /private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:179:16: error: use of undeclared identifier 'ret'
              return ret;
                     ^
      /private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:185:8: error: unknown type name 'rd_kafka_AclBindingFilter_t'
      static rd_kafka_AclBindingFilter_t*
             ^
      /private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:192:9: error: use of undeclared identifier 'rd_kafka_AclBindingFilter_t'
              rd_kafka_AclBindingFilter_t* ret = NULL;
              ^
      /private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:192:38: error: use of undeclared identifier 'ret'
              rd_kafka_AclBindingFilter_t* ret = NULL;
                                           ^
      /private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:201:21: error: use of undeclared identifier 'ret'
                          ret = rd_kafka_AclBindingFilter_new(restype, resname, \
                          ^
      /private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:201:27: error: implicit declaration of function 'rd_kafka_AclBindingFilter_new' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
                          ret = rd_kafka_AclBindingFilter_new(restype, resname, \
                                ^
      /private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:208:16: error: use of undeclared identifier 'ret'
              return ret;
                     ^
      /private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:1045:9: error: use of undeclared identifier 'rd_kafka_AclBinding_t'
              rd_kafka_AclBinding_t **c_objs = NULL;
              ^
      /private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:1045:33: error: use of undeclared identifier 'c_objs'
              rd_kafka_AclBinding_t **c_objs = NULL;
                                      ^
      /private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:1081:46: error: use of undeclared identifier 'RD_KAFKA_ADMIN_OP_CREATEACLS'; did you mean 'RD_KAFKA_ADMIN_OP_CREATETOPICS'?
              c_options = Admin_options_to_c(self, RD_KAFKA_ADMIN_OP_CREATEACLS,
                                                   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
                                                   RD_KAFKA_ADMIN_OP_CREATETOPICS
      /usr/local/include/librdkafka/rdkafka.h:6117:9: note: 'RD_KAFKA_ADMIN_OP_CREATETOPICS' declared here
              RD_KAFKA_ADMIN_OP_CREATETOPICS,     /**< CreateTopics */
              ^
      /private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:1095:9: error: use of undeclared identifier 'c_objs'
              c_objs = malloc(sizeof(*c_objs) * cnt);
              ^
      /private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:1095:33: error: use of undeclared identifier 'c_objs'
              c_objs = malloc(sizeof(*c_objs) * cnt);
                                      ^
      /private/var/folders/wr/mr29lwrx0hb8gk27c1xp31l80000gn/T/pip-install-gy0dfrg3/confluent-kafka_f8523e7803884be9823241ff555d6b98/src/confluent_kafka/src/Admin.c:1112:17: error: use of undeclared identifier 'c_objs'
                      c_objs[i] = Admin_py_to_c_AclBinding(res, errstr, sizeof(errstr));
                      ^
      fatal error: too many errors emitted, stopping now [-ferror-limit=]
      20 errors generated.
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> confluent-kafka

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.
WARNING: There was an error checking the latest version of pip.

Bolun Huang · Answer 23 · Sat Jun 25 2022 10:58:20 GMT+0800 (China Standard Time)

@jliunyu I figured this out.

On Apple M1 hosts, all its needed is this:

CPATH=/opt/homebrew/Cellar/librdkafka/1.8.2/include pip install confluent-kafka

Thanks for the suggestion, @armenzg.

It does find the librdkafka by doing this; However, I have got an subsequent error library not found for -lrdkafka:

clang -bundle -undefined dynamic_lookup -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -L/opt/homebrew/opt/icu4c/lib -I/opt/homebrew/opt/icu4c/include build/temp.macosx-12-arm64-cpython-39/private/var/folders/1k/7cx1mxyn7zg_r0sj4m64vlyc0000gn/T/pip-install-v_24bcxa/confluent-kafka_c9cbbfa9cbed4ab9a463c5cc0a4ad715/src/confluent_kafka/src/Admin.o build/temp.macosx-12-arm64-cpython-39/private/var/folders/1k/7cx1mxyn7zg_r0sj4m64vlyc0000gn/T/pip-install-v_24bcxa/confluent-kafka_c9cbbfa9cbed4ab9a463c5cc0a4ad715/src/confluent_kafka/src/AdminTypes.o build/temp.macosx-12-arm64-cpython-39/private/var/folders/1k/7cx1mxyn7zg_r0sj4m64vlyc0000gn/T/pip-install-v_24bcxa/confluent-kafka_c9cbbfa9cbed4ab9a463c5cc0a4ad715/src/confluent_kafka/src/Consumer.o build/temp.macosx-12-arm64-cpython-39/private/var/folders/1k/7cx1mxyn7zg_r0sj4m64vlyc0000gn/T/pip-install-v_24bcxa/confluent-kafka_c9cbbfa9cbed4ab9a463c5cc0a4ad715/src/confluent_kafka/src/Metadata.o build/temp.macosx-12-arm64-cpython-39/private/var/folders/1k/7cx1mxyn7zg_r0sj4m64vlyc0000gn/T/pip-install-v_24bcxa/confluent-kafka_c9cbbfa9cbed4ab9a463c5cc0a4ad715/src/confluent_kafka/src/Producer.o build/temp.macosx-12-arm64-cpython-39/private/var/folders/1k/7cx1mxyn7zg_r0sj4m64vlyc0000gn/T/pip-install-v_24bcxa/confluent-kafka_c9cbbfa9cbed4ab9a463c5cc0a4ad715/src/confluent_kafka/src/confluent_kafka.o -lrdkafka -o build/lib.macosx-12-arm64-cpython-39/confluent_kafka/cimpl.cpython-39-darwin.so ld: library not found for -lrdkafka clang: error: linker command failed with exit code 1 (use -v to see invocation) error: command '/usr/bin/clang' failed with exit code 1 [end of output]

env setup:

librdkafka v1.9.0
confluent-kafka v1.8.2
python v3.9

Any suggestion?

Venkatesh Gopalakrishnan · Answer 24 · Mon Jun 27 2022 19:16:29 GMT+0800 (China Standard Time)

If you put the lines in the shell script file such as .zshrc and .bashrc, you should use them with export:
export C_INCLUDE_PATH=/opt/homebrew/Cellar/librdkafka/1.8.2/include
export LIBRARY_PATH=/opt/homebrew/Cellar/librdkafka/1.8.2/lib

This worked for me!

madisonleavo · Answer 25 · Tue Jun 28 2022 22:51:04 GMT+0800 (China Standard Time)

Take care, sometimes librdkafka and confluent-kafka-python versions don't match.

error: "confluent-kafka-python requires librdkafka v1.9.0 or later. Install the latest version of librdkafka from Homebrew by running brew install librdkafka or brew upgrade librdkafka"

Once I brew upgrade librdkafka then I set the two PATH vars with 1.9.0 version. Then pip install confluent-kafka works properly.

@ddbhuang maybe this addresses your issue?

Tal Vaizman · Answer 26 · Wed Jul 06 2022 06:22:10 GMT+0800 (China Standard Time)

Apparently, pip has a way to define these variables for a single installation.
This should work for whoever is trying to install using pip and it is much cleaner then setting env vars IMO:

pip install --global-option=build_ext --global-option="-L/opt/homebrew/Cellar/librdkafka/1.9.0/lib" --global-option="-I/opt/homebrew/Cellar/librdkafka/1.9.0/include" confluent-kafka

Just make sure to change your librdkafka version according to what's installed on your machine.

Carlos Tangerino · Answer 27 · Wed Jul 06 2022 19:34:23 GMT+0800 (China Standard Time)

Same error for me. I'll abandon this library :(

Gregor Karetka · Answer 28 · Thu Jul 07 2022 20:49:52 GMT+0800 (China Standard Time)

Apparently, pip has a way to define these variables for a single installation. This should work for whoever is trying to install using pip and it is much cleaner then setting env vars IMO:
pip install --global-option=build_ext --global-option="-L/opt/homebrew/Cellar/librdkafka/1.9.0/lib" --global-option="-I/opt/homebrew/Cellar/librdkafka/1.9.0/include" confluent-kafka
Just make sure to change your librdkafka version according to what's installed on your machine.

In order this to work properly version of the library must be the same as in these vars. The newest version is 1.9.1 so this won't work. Run brew info librdkafka in order to determine version of library.

Chris · Answer 29 · Fri Jul 15 2022 15:12:43 GMT+0800 (China Standard Time)

Most likely your library and cpath are not set for brew, where you probably installed kcat or librdkafka.
Just add all the brew libs etc to your .zshrc:

export C_INCLUDE_PATH=$(brew --prefix)/include
export LIBRARY_PATH=$(brew --prefix)/lib

Eduardo Gutiérrez Silva · Answer 30 · Wed Jul 27 2022 04:34:00 GMT+0800 (China Standard Time)

Confirming working steps in an M1 machine running macOS 12.4 and Python 3.10.5 with brew installed librdkafka.

brew upgrade
brew install librdkafka

export C_INCLUDE_PATH=$(brew --prefix)/include
export LIBRARY_PATH=$(brew --prefix)/lib

pip install confluent-kafka

Thanks everyone for the help!

Pranav Rathi · Answer 31 · Tue Feb 27 2024 21:34:55 GMT+0800 (China Standard Time)

Thanks for all the great answers.

Closing this issue as wheels for MacOS arm64 were added as part of v1.9.2 release and linux arm64 wheels were added in v2.1.0 release.