Librabbitmq fails to install in python 3.7 on debian buster when make is in path
potiuk opened this issue · comments
When make can be found in the path, librabbitmq fails to install in python3.7 variant on debian buster. It tries to recompile librabbitmq-c and fails. When make is not found in path, it installs correctly from wheels.
Environment:
Docker image python:3.7-slim-buster
This works fine:
3z [jarek:~] [airflow-3.6] 127 % docker run --entrypoint bash -it python:3.7-slim-buster
root@42dab7abf345:/# pip install librabbitmq
Collecting librabbitmq
Downloading librabbitmq-2.0.0.tar.gz (42 kB)
|████████████████████████████████| 42 kB 667 kB/s
Collecting amqp>=1.4.6
Downloading amqp-2.5.2-py2.py3-none-any.whl (49 kB)
|████████████████████████████████| 49 kB 4.0 MB/s
Collecting six>=1.0.0
Downloading six-1.14.0-py2.py3-none-any.whl (10 kB)
Collecting vine<5.0.0a1,>=1.1.3
Downloading vine-1.3.0-py2.py3-none-any.whl (14 kB)
Building wheels for collected packages: librabbitmq
Building wheel for librabbitmq (setup.py) ... done
Created wheel for librabbitmq: filename=librabbitmq-2.0.0-py3-none-any.whl size=18193 sha256=3aec6d28544fc550402562f48a4beb1905354754f16883c33ff60fdd515319a9
Stored in directory: /root/.cache/pip/wheels/da/77/bf/93ad3b74b6801c51fcd184fe6b330e842b7c73810140463c04
Successfully built librabbitmq
Installing collected packages: vine, amqp, six, librabbitmq
Successfully installed amqp-2.5.2 librabbitmq-2.0.0 six-1.14.0 vine-1.3.0
root@42dab7abf345:/#
This doesn't:
3z [jarek:~] [airflow-3.6] 48s 1 % docker run --entrypoint bash -it python:3.7-slim-buster
root@a8edcaf9a7eb:/# apt update
Get:1 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:2 http://deb.debian.org/debian buster InRelease [121 kB]
Get:3 http://deb.debian.org/debian buster-updates InRelease [49.3 kB]
Get:4 http://security.debian.org/debian-security buster/updates/main amd64 Packages [197 kB]
Get:5 http://deb.debian.org/debian buster/main amd64 Packages [7905 kB]
Get:6 http://deb.debian.org/debian buster-updates/main amd64 Packages [7380 B]
Fetched 8345 kB in 2s (4329 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
4 packages can be upgraded. Run 'apt list --upgradable' to see them.
root@a8edcaf9a7eb:/# apt install build-essential git
..... lots of output .....
root@a8edcaf9a7eb:/# pip install librabbitmq
Collecting librabbitmq
Downloading librabbitmq-2.0.0.tar.gz (42 kB)
|████████████████████████████████| 42 kB 768 kB/s
Collecting amqp>=1.4.6
Downloading amqp-2.5.2-py2.py3-none-any.whl (49 kB)
|████████████████████████████████| 49 kB 3.6 MB/s
Collecting six>=1.0.0
Downloading six-1.14.0-py2.py3-none-any.whl (10 kB)
Collecting vine<5.0.0a1,>=1.1.3
Downloading vine-1.3.0-py2.py3-none-any.whl (14 kB)
Building wheels for collected packages: librabbitmq
Building wheel for librabbitmq (setup.py) ... error
ERROR: Command errored out with exit status 1:
command: /usr/local/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-511qt4su/librabbitmq/setup.py'"'"'; __file__='"'"'/tmp/pip-install-511qt4su/librabbitmq/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-utcex30b
cwd: /tmp/pip-install-511qt4su/librabbitmq/
Complete output (109 lines):
/tmp/pip-install-511qt4su/librabbitmq/setup.py:167: DeprecationWarning: 'U' mode is deprecated
long_description = open(os.path.join(BASE_PATH, 'README.rst'), 'U').read()
running build
- pull submodule rabbitmq-c...
Cloning into 'rabbitmq-c'...
Note: checking out 'caad0ef1533783729c7644a226c989c79b4c497b'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b <new-branch-name>
- autoreconf
sh: 1: autoreconf: not found
- configure rabbitmq-c...
/bin/sh: 0: Can't open configure
running bdist_wheel
installing to build/bdist.linux-x86_64/wheel
running install
running install_lib
running build_py
creating build
creating build/lib.linux-x86_64-3.7
creating build/lib.linux-x86_64-3.7/librabbitmq
copying librabbitmq/__init__.py -> build/lib.linux-x86_64-3.7/librabbitmq
creating build/lib.linux-x86_64-3.7/funtests
copying funtests/disabled_basic_get_leak.py -> build/lib.linux-x86_64-3.7/funtests
copying funtests/__init__.py -> build/lib.linux-x86_64-3.7/funtests
copying funtests/config.py -> build/lib.linux-x86_64-3.7/funtests
copying funtests/setup.py -> build/lib.linux-x86_64-3.7/funtests
creating build/lib.linux-x86_64-3.7/funtests/tests
copying funtests/tests/test_channel_errors.py -> build/lib.linux-x86_64-3.7/funtests/tests
copying funtests/tests/__init__.py -> build/lib.linux-x86_64-3.7/funtests/tests
running build_ext
building '_librabbitmq' extension
creating build/temp.linux-x86_64-3.7
creating build/temp.linux-x86_64-3.7/tmp
creating build/temp.linux-x86_64-3.7/tmp/pip-install-511qt4su
creating build/temp.linux-x86_64-3.7/tmp/pip-install-511qt4su/librabbitmq
creating build/temp.linux-x86_64-3.7/tmp/pip-install-511qt4su/librabbitmq/Modules
creating build/temp.linux-x86_64-3.7/tmp/pip-install-511qt4su/librabbitmq/Modules/_librabbitmq
creating build/temp.linux-x86_64-3.7/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c
creating build/temp.linux-x86_64-3.7/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq -I/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c -I/usr/local/include/python3.7m -c /tmp/pip-install-511qt4su/librabbitmq/Modules/_librabbitmq/connection.c -o build/temp.linux-x86_64-3.7/tmp/pip-install-511qt4su/librabbitmq/Modules/_librabbitmq/connection.o
/tmp/pip-install-511qt4su/librabbitmq/Modules/_librabbitmq/connection.c: In function ‘PyDict_ToAMQTable’:
/tmp/pip-install-511qt4su/librabbitmq/Modules/_librabbitmq/connection.c:299:29: warning: comparison of integer expressions of different signedness: ‘uint64_t’ {aka ‘long unsigned int’} and ‘int’ [-Wsign-compare]
if (clong_value == -1)
^~
/tmp/pip-install-511qt4su/librabbitmq/Modules/_librabbitmq/connection.c: In function ‘PyIter_ToAMQArray’:
/tmp/pip-install-511qt4su/librabbitmq/Modules/_librabbitmq/connection.c:365:12: warning: suggest parentheses around assignment used as truth value [-Wparentheses]
while (item = PyIter_Next(iterator)) {
^~~~
/tmp/pip-install-511qt4su/librabbitmq/Modules/_librabbitmq/connection.c:410:1: warning: label ‘error’ defined but not used [-Wunused-label]
error:
^~~~~
/tmp/pip-install-511qt4su/librabbitmq/Modules/_librabbitmq/connection.c: In function ‘PyRabbitMQ_ConnectionType_init’:
/tmp/pip-install-511qt4su/librabbitmq/Modules/_librabbitmq/connection.c:1003:16: warning: returning ‘PyObject *’ {aka ‘struct _object *’} from a function with return type ‘int’ makes integer from pointer without a cast [-Wint-conversion]
return PyErr_NoMemory();
^~~~~~~~~~~~~~~~
/tmp/pip-install-511qt4su/librabbitmq/Modules/_librabbitmq/connection.c: In function ‘PyRabbitMQ_Connection_close’:
/tmp/pip-install-511qt4su/librabbitmq/Modules/_librabbitmq/connection.c:1117:22: warning: variable ‘reply’ set but not used [-Wunused-but-set-variable]
amqp_rpc_reply_t reply;
^~~~~
/tmp/pip-install-511qt4su/librabbitmq/Modules/_librabbitmq/connection.c: In function ‘PyRabbitMQ_recv’:
/tmp/pip-install-511qt4su/librabbitmq/Modules/_librabbitmq/connection.c:1359:13: warning: implicit declaration of function ‘amqp_simple_wait_frame_on_channel’; did you mean ‘amqp_simple_wait_frame_noblock’? [-Wimplicit-function-declaration]
amqp_simple_wait_frame_on_channel(conn, cur_channel, &frame);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
amqp_simple_wait_frame_noblock
/tmp/pip-install-511qt4su/librabbitmq/Modules/_librabbitmq/connection.c: In function ‘PyRabbitMQ_Connection_basic_cancel’:
/tmp/pip-install-511qt4su/librabbitmq/Modules/_librabbitmq/connection.c:1949:29: warning: variable ‘ok’ set but not used [-Wunused-but-set-variable]
amqp_basic_cancel_ok_t *ok;
^~
/tmp/pip-install-511qt4su/librabbitmq/Modules/_librabbitmq/connection.c: In function ‘PyRabbitMQ_Connection_flow’:
/tmp/pip-install-511qt4su/librabbitmq/Modules/_librabbitmq/connection.c:2074:29: warning: variable ‘ok’ set but not used [-Wunused-but-set-variable]
amqp_channel_flow_ok_t *ok;
^~
/tmp/pip-install-511qt4su/librabbitmq/Modules/_librabbitmq/connection.c: In function ‘PyRabbitMQ_Connection_basic_recover’:
/tmp/pip-install-511qt4su/librabbitmq/Modules/_librabbitmq/connection.c:2107:30: warning: variable ‘ok’ set but not used [-Wunused-but-set-variable]
amqp_basic_recover_ok_t *ok;
^~
/tmp/pip-install-511qt4su/librabbitmq/Modules/_librabbitmq/connection.c: At top level:
/tmp/pip-install-511qt4su/librabbitmq/Modules/_librabbitmq/connection.c:526:17: warning: ‘AMQTable_toPyDict’ is static but used in inline function ‘basic_properties_to_PyDict’ which is not static
value = AMQTable_toPyDict(&(props->headers));
^~~~~~~~~~~~~~~~~
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq -I/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c -I/usr/local/include/python3.7m -c /tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq/amqp_api.c -o build/temp.linux-x86_64-3.7/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq/amqp_api.o
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq -I/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c -I/usr/local/include/python3.7m -c /tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq/amqp_connection.c -o build/temp.linux-x86_64-3.7/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq/amqp_connection.o
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq -I/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c -I/usr/local/include/python3.7m -c /tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq/amqp_consumer.c -o build/temp.linux-x86_64-3.7/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq/amqp_consumer.o
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq -I/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c -I/usr/local/include/python3.7m -c /tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq/amqp_framing.c -o build/temp.linux-x86_64-3.7/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq/amqp_framing.o
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq -I/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c -I/usr/local/include/python3.7m -c /tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq/amqp_hostcheck.c -o build/temp.linux-x86_64-3.7/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq/amqp_hostcheck.o
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq -I/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c -I/usr/local/include/python3.7m -c /tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq/amqp_mem.c -o build/temp.linux-x86_64-3.7/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq/amqp_mem.o
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq -I/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c -I/usr/local/include/python3.7m -c /tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq/amqp_socket.c -o build/temp.linux-x86_64-3.7/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq/amqp_socket.o
/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq/amqp_socket.c: In function ‘amqp_poll’:
/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq/amqp_socket.c:355:3: error: #error "poll() or select() is needed to compile rabbitmq-c"
# error "poll() or select() is needed to compile rabbitmq-c"
^~~~~
/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq/amqp_socket.c: In function ‘amqp_login_inner’:
/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq/amqp_socket.c:1334:53: error: ‘AMQ_PLATFORM’ undeclared (first use in this function)
amqp_table_construct_utf8_entry("platform", AMQ_PLATFORM);
^~~~~~~~~~~~
/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq/amqp_socket.c:1334:53: note: each undeclared identifier is reported only once for each function it appears in
/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq/amqp_socket.c: In function ‘amqp_poll’:
/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq/amqp_socket.c:357:1: warning: control reaches end of non-void function [-Wreturn-type]
}
^
error: command 'gcc' failed with exit status 1
----------------------------------------
ERROR: Failed building wheel for librabbitmq
Running setup.py clean for librabbitmq
Failed to build librabbitmq
Installing collected packages: vine, amqp, six, librabbitmq
Running setup.py install for librabbitmq ... error
ERROR: Command errored out with exit status 1:
command: /usr/local/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-511qt4su/librabbitmq/setup.py'"'"'; __file__='"'"'/tmp/pip-install-511qt4su/librabbitmq/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-lsqagyp_/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python3.7m/librabbitmq
cwd: /tmp/pip-install-511qt4su/librabbitmq/
Complete output (94 lines):
/tmp/pip-install-511qt4su/librabbitmq/setup.py:167: DeprecationWarning: 'U' mode is deprecated
long_description = open(os.path.join(BASE_PATH, 'README.rst'), 'U').read()
running build
- autoreconf
sh: 1: autoreconf: not found
- configure rabbitmq-c...
/bin/sh: 0: Can't open configure
running install
running install_lib
running build_py
creating build
creating build/lib.linux-x86_64-3.7
creating build/lib.linux-x86_64-3.7/librabbitmq
copying librabbitmq/__init__.py -> build/lib.linux-x86_64-3.7/librabbitmq
creating build/lib.linux-x86_64-3.7/funtests
copying funtests/disabled_basic_get_leak.py -> build/lib.linux-x86_64-3.7/funtests
copying funtests/__init__.py -> build/lib.linux-x86_64-3.7/funtests
copying funtests/config.py -> build/lib.linux-x86_64-3.7/funtests
copying funtests/setup.py -> build/lib.linux-x86_64-3.7/funtests
creating build/lib.linux-x86_64-3.7/funtests/tests
copying funtests/tests/test_channel_errors.py -> build/lib.linux-x86_64-3.7/funtests/tests
copying funtests/tests/__init__.py -> build/lib.linux-x86_64-3.7/funtests/tests
running build_ext
building '_librabbitmq' extension
creating build/temp.linux-x86_64-3.7
creating build/temp.linux-x86_64-3.7/tmp
creating build/temp.linux-x86_64-3.7/tmp/pip-install-511qt4su
creating build/temp.linux-x86_64-3.7/tmp/pip-install-511qt4su/librabbitmq
creating build/temp.linux-x86_64-3.7/tmp/pip-install-511qt4su/librabbitmq/Modules
creating build/temp.linux-x86_64-3.7/tmp/pip-install-511qt4su/librabbitmq/Modules/_librabbitmq
creating build/temp.linux-x86_64-3.7/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c
creating build/temp.linux-x86_64-3.7/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq -I/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c -I/usr/local/include/python3.7m -c /tmp/pip-install-511qt4su/librabbitmq/Modules/_librabbitmq/connection.c -o build/temp.linux-x86_64-3.7/tmp/pip-install-511qt4su/librabbitmq/Modules/_librabbitmq/connection.o
/tmp/pip-install-511qt4su/librabbitmq/Modules/_librabbitmq/connection.c: In function ‘PyDict_ToAMQTable’:
/tmp/pip-install-511qt4su/librabbitmq/Modules/_librabbitmq/connection.c:299:29: warning: comparison of integer expressions of different signedness: ‘uint64_t’ {aka ‘long unsigned int’} and ‘int’ [-Wsign-compare]
if (clong_value == -1)
^~
/tmp/pip-install-511qt4su/librabbitmq/Modules/_librabbitmq/connection.c: In function ‘PyIter_ToAMQArray’:
/tmp/pip-install-511qt4su/librabbitmq/Modules/_librabbitmq/connection.c:365:12: warning: suggest parentheses around assignment used as truth value [-Wparentheses]
while (item = PyIter_Next(iterator)) {
^~~~
/tmp/pip-install-511qt4su/librabbitmq/Modules/_librabbitmq/connection.c:410:1: warning: label ‘error’ defined but not used [-Wunused-label]
error:
^~~~~
/tmp/pip-install-511qt4su/librabbitmq/Modules/_librabbitmq/connection.c: In function ‘PyRabbitMQ_ConnectionType_init’:
/tmp/pip-install-511qt4su/librabbitmq/Modules/_librabbitmq/connection.c:1003:16: warning: returning ‘PyObject *’ {aka ‘struct _object *’} from a function with return type ‘int’ makes integer from pointer without a cast [-Wint-conversion]
return PyErr_NoMemory();
^~~~~~~~~~~~~~~~
/tmp/pip-install-511qt4su/librabbitmq/Modules/_librabbitmq/connection.c: In function ‘PyRabbitMQ_Connection_close’:
/tmp/pip-install-511qt4su/librabbitmq/Modules/_librabbitmq/connection.c:1117:22: warning: variable ‘reply’ set but not used [-Wunused-but-set-variable]
amqp_rpc_reply_t reply;
^~~~~
/tmp/pip-install-511qt4su/librabbitmq/Modules/_librabbitmq/connection.c: In function ‘PyRabbitMQ_recv’:
/tmp/pip-install-511qt4su/librabbitmq/Modules/_librabbitmq/connection.c:1359:13: warning: implicit declaration of function ‘amqp_simple_wait_frame_on_channel’; did you mean ‘amqp_simple_wait_frame_noblock’? [-Wimplicit-function-declaration]
amqp_simple_wait_frame_on_channel(conn, cur_channel, &frame);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
amqp_simple_wait_frame_noblock
/tmp/pip-install-511qt4su/librabbitmq/Modules/_librabbitmq/connection.c: In function ‘PyRabbitMQ_Connection_basic_cancel’:
/tmp/pip-install-511qt4su/librabbitmq/Modules/_librabbitmq/connection.c:1949:29: warning: variable ‘ok’ set but not used [-Wunused-but-set-variable]
amqp_basic_cancel_ok_t *ok;
^~
/tmp/pip-install-511qt4su/librabbitmq/Modules/_librabbitmq/connection.c: In function ‘PyRabbitMQ_Connection_flow’:
/tmp/pip-install-511qt4su/librabbitmq/Modules/_librabbitmq/connection.c:2074:29: warning: variable ‘ok’ set but not used [-Wunused-but-set-variable]
amqp_channel_flow_ok_t *ok;
^~
/tmp/pip-install-511qt4su/librabbitmq/Modules/_librabbitmq/connection.c: In function ‘PyRabbitMQ_Connection_basic_recover’:
/tmp/pip-install-511qt4su/librabbitmq/Modules/_librabbitmq/connection.c:2107:30: warning: variable ‘ok’ set but not used [-Wunused-but-set-variable]
amqp_basic_recover_ok_t *ok;
^~
/tmp/pip-install-511qt4su/librabbitmq/Modules/_librabbitmq/connection.c: At top level:
/tmp/pip-install-511qt4su/librabbitmq/Modules/_librabbitmq/connection.c:526:17: warning: ‘AMQTable_toPyDict’ is static but used in inline function ‘basic_properties_to_PyDict’ which is not static
value = AMQTable_toPyDict(&(props->headers));
^~~~~~~~~~~~~~~~~
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq -I/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c -I/usr/local/include/python3.7m -c /tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq/amqp_api.c -o build/temp.linux-x86_64-3.7/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq/amqp_api.o
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq -I/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c -I/usr/local/include/python3.7m -c /tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq/amqp_connection.c -o build/temp.linux-x86_64-3.7/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq/amqp_connection.o
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq -I/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c -I/usr/local/include/python3.7m -c /tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq/amqp_consumer.c -o build/temp.linux-x86_64-3.7/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq/amqp_consumer.o
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq -I/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c -I/usr/local/include/python3.7m -c /tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq/amqp_framing.c -o build/temp.linux-x86_64-3.7/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq/amqp_framing.o
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq -I/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c -I/usr/local/include/python3.7m -c /tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq/amqp_hostcheck.c -o build/temp.linux-x86_64-3.7/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq/amqp_hostcheck.o
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq -I/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c -I/usr/local/include/python3.7m -c /tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq/amqp_mem.c -o build/temp.linux-x86_64-3.7/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq/amqp_mem.o
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq -I/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c -I/usr/local/include/python3.7m -c /tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq/amqp_socket.c -o build/temp.linux-x86_64-3.7/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq/amqp_socket.o
/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq/amqp_socket.c: In function ‘amqp_poll’:
/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq/amqp_socket.c:355:3: error: #error "poll() or select() is needed to compile rabbitmq-c"
# error "poll() or select() is needed to compile rabbitmq-c"
^~~~~
/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq/amqp_socket.c: In function ‘amqp_login_inner’:
/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq/amqp_socket.c:1334:53: error: ‘AMQ_PLATFORM’ undeclared (first use in this function)
amqp_table_construct_utf8_entry("platform", AMQ_PLATFORM);
^~~~~~~~~~~~
/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq/amqp_socket.c:1334:53: note: each undeclared identifier is reported only once for each function it appears in
/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq/amqp_socket.c: In function ‘amqp_poll’:
/tmp/pip-install-511qt4su/librabbitmq/rabbitmq-c/librabbitmq/amqp_socket.c:357:1: warning: control reaches end of non-void function [-Wreturn-type]
}
^
error: command 'gcc' failed with exit status 1
----------------------------------------
ERROR: Command errored out with exit status 1: /usr/local/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-511qt4su/librabbitmq/setup.py'"'"'; __file__='"'"'/tmp/pip-install-511qt4su/librabbitmq/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-lsqagyp_/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python3.7m/librabbitmq Check the logs for full command output.
root@a8edcaf9a7eb:/#
I found that I can still correctly install librabbitmq by renaming /usr/bin/make temporarily
Note that in python:3.6-slim-buster image the same sequence of commands leads to successfully installing librabbitmq from wheels without recompiling it.
root@488db7602d26:/# pip install librabbitmq
Collecting librabbitmq
Downloading librabbitmq-2.0.0-cp36-cp36m-manylinux1_x86_64.whl (218 kB)
|████████████████████████████████| 218 kB 1.2 MB/s
Collecting six>=1.0.0
Downloading six-1.14.0-py2.py3-none-any.whl (10 kB)
Collecting amqp>=1.4.6
Downloading amqp-2.5.2-py2.py3-none-any.whl (49 kB)
|████████████████████████████████| 49 kB 6.6 MB/s
Collecting vine<5.0.0a1,>=1.1.3
Downloading vine-1.3.0-py2.py3-none-any.whl (14 kB)
Installing collected packages: six, vine, amqp, librabbitmq
Successfully installed amqp-2.5.2 librabbitmq-2.0.0 six-1.14.0 vine-1.3.0
root@488db7602d26:/#
I presume the problem is that there are no pre-compiled py37 and above wheels for librabbitmq and then it tries to compile it on its own.
@auvipy Can you help with it? This is related to Apache Airflow.
first of all, we do not recommend this package and instead focused on improving py-amqp with cython optimisation :)
I want to know what advantage this package is providing apache airflow over py-amqp?
I suspect that there is no advantage except that we currently have this library added to the dependency and all tests for Celera use it. What does the migration from this library to py-amqp look like for Celery?
I think I found a solution. Is it enough that we will not install this library and Celery will use an alternative fully compatible?
https://github.com/celery/kombu/blob/81101877a09e8f4442555c6c74b4836f113e4f51/kombu/connection.py#L250-L252
actually py-amqp has been the main supported library for last two years. our all focus is on py-amqp. imho py-amqp is more feature rich. i would suggest drop librabbitmq as dependency and focus your effort to py-amqp along with the celery core team :) you can also check the recently merge cython improvements on py-amqp package :)
just a quick one - Is this a dop-in replacement for librabbitmq ? Or should we re-write our rabbitmq integration to use a new API?
this is a drop in replacement. I would try py-amqp docs first
FYI - I m asking because we have like 200 integrations in Airflow some of them written by someone 2 years ago, so we do not know all the ins-outs of all the integrations :). But will take a look at the libs
first of all, just try py-amqp and see how many tests are passing. you can also mention me on the PR. or share the tests with me. not much should be different. we can identify which API/tests need adjustment
Thanks @auvipy for your offer to help! All good. I replaced librabbitmq exlicitly with amqp, rebuilt the image and run our integration tests with rabbitmq . All passing :).
ests/executors/test_base_executor.py::TestBaseExecutor::test_gauge_executor_metrics SKIPPED [ 2%]
tests/executors/test_base_executor.py::TestBaseExecutor::test_get_event_buffer SKIPPED [ 4%]
tests/executors/test_celery_executor.py::TestCeleryExecutor::test_celery_integration_0_amqp_guest_guest_rabbitmq_5672 PASSED [ 6%]
tests/executors/test_celery_executor.py::TestCeleryExecutor::test_celery_integration_1_redis_redis_6379_0 PASSED [ 9%]
tests/executors/test_celery_executor.py::TestCeleryExecutor::test_error_sending_task PASSED [ 11%]
tests/executors/test_celery_executor.py::TestCeleryExecutor::test_exception_propagation SKIPPED [ 13%]
tests/executors/test_celery_executor.py::TestCeleryExecutor::test_gauge_executor_metrics SKIPPED [ 15%]
tests/executors/test_celery_executor.py::test_operation_timeout_config SKIPPED [ 18%]
tests/executors/test_celery_executor.py::TestBulkStateFetcher::test_should_support_base_backend PASSED [ 20%]
tests/executors/test_celery_executor.py::TestBulkStateFetcher::test_should_support_db_backend PASSED [ 22%]
tests/executors/test_celery_executor.py::TestBulkStateFetcher::test_should_support_kv_backend PASSED [ 25%]
tests/executors/test_dask_executor.py::TestDaskExecutor::test_backfill_integration SKIPPED [ 27%]
tests/executors/test_dask_executor.py::TestDaskExecutor::test_dask_executor_functions SKIPPED [ 29%]
tests/executors/test_dask_executor.py::TestDaskExecutorTLS::test_gauge_executor_metrics SKIPPED [ 31%]
tests/executors/test_dask_executor.py::TestDaskExecutorTLS::test_tls SKIPPED [ 34%]
tests/executors/test_debug_executor.py::TestDebugExecutor::test_sync SKIPPED [ 36%]
tests/executors/test_debug_executor.py::TestDebugExecutor::test_run_task SKIPPED [ 38%]
tests/executors/test_debug_executor.py::TestDebugExecutor::test_queue_task_instance SKIPPED [ 40%]
tests/executors/test_debug_executor.py::TestDebugExecutor::test_trigger_tasks SKIPPED [ 43%]
tests/executors/test_debug_executor.py::TestDebugExecutor::test_end SKIPPED [ 45%]
tests/executors/test_debug_executor.py::TestDebugExecutor::test_fail_fast SKIPPED [ 47%]
tests/executors/test_executor_loader.py::TestExecutorLoader::test_should_support_custom_path SKIPPED [ 50%]
tests/executors/test_executor_loader.py::TestExecutorLoader::test_should_support_executor_from_core_0_LocalExecutor SKIPPED [ 52%]
tests/executors/test_executor_loader.py::TestExecutorLoader::test_should_support_executor_from_core_1_DebugExecutor SKIPPED [ 54%]
tests/executors/test_executor_loader.py::TestExecutorLoader::test_should_support_plugins SKIPPED [ 56%]
tests/executors/test_kubernetes_executor.py::TestAirflowKubernetesScheduler::test_create_pod_id SKIPPED [ 59%]
tests/executors/test_kubernetes_executor.py::TestAirflowKubernetesScheduler::test_execution_date_serialize_deserialize SKIPPED [ 61%]
tests/executors/test_kubernetes_executor.py::TestAirflowKubernetesScheduler::test_make_safe_label_value SKIPPED [ 63%]
tests/executors/test_kubernetes_executor.py::TestKubeConfig::test_kube_config_git_sync_run_as_user_empty_string SKIPPED [ 65%]
tests/executors/test_kubernetes_executor.py::TestKubeConfig::test_kube_config_git_sync_run_as_user_not_present SKIPPED [ 68%]
tests/executors/test_kubernetes_executor.py::TestKubeConfig::test_kube_config_git_sync_run_as_user_root SKIPPED [ 70%]
tests/executors/test_kubernetes_executor.py::TestKubeConfig::test_kube_config_no_worker_annotations SKIPPED [ 72%]
tests/executors/test_kubernetes_executor.py::TestKubeConfig::test_kube_config_worker_annotations_properly_parsed SKIPPED [ 75%]
tests/executors/test_kubernetes_executor.py::TestKubernetesExecutor::test_change_state_failed_no_deletion SKIPPED [ 77%]
tests/executors/test_kubernetes_executor.py::TestKubernetesExecutor::test_change_state_failed_pod_deletion SKIPPED [ 79%]
tests/executors/test_kubernetes_executor.py::TestKubernetesExecutor::test_change_state_running SKIPPED [ 81%]
tests/executors/test_kubernetes_executor.py::TestKubernetesExecutor::test_change_state_skip_pod_deletion SKIPPED [ 84%]
tests/executors/test_kubernetes_executor.py::TestKubernetesExecutor::test_change_state_success SKIPPED [ 86%]
tests/executors/test_kubernetes_executor.py::TestKubernetesExecutor::test_gauge_executor_metrics SKIPPED [ 88%]
tests/executors/test_kubernetes_executor.py::TestKubernetesExecutor::test_run_next_exception SKIPPED [ 90%]
tests/executors/test_local_executor.py::TestLocalExecutor::test_execution_limited_parallelism SKIPPED [ 93%]
tests/executors/test_local_executor.py::TestLocalExecutor::test_execution_unlimited_parallelism SKIPPED [ 95%]
tests/executors/test_local_executor.py::TestLocalExecutor::test_gauge_executor_metrics SKIPPED [ 97%]
tests/executors/test_sequential_executor.py::TestSequentialExecutor::test_gauge_executor_metrics SKIPPED [100%]
6 passed, 38 skipped, 1 warning in 47.65s
hell yeah!
this is a drop in replacement. I would try py-amqp docs first
I wish the docs would say this. Currently the Celery/Kombu docs state:
AMQP transport using the py-amqp, librabbitmq, or qpid-python libraries.
High performance AMQP transport written in C - when using librabbitmq
This is automatically enabled if librabbitmq is installed:
$ pip install librabbitmq
This does not imply to me in any way that py-amqp is also high-performance (I think that multiple sources state it is native Python suggests even the opposite) nor that I can literally just switch my requirements file from librabbitmq to amqp and everything would work... and in fact, it did! I only tried this after several hours of trying to install librabbitmq though.
Appreciate all the work you guys do! The docs have usually been good, but I do think this is a gap.
Feel free to contribute any documentation improvements you think are necessary.