Unable to install fleetspeak-client-bin (0.1.11) on ppc64le
helix-loop opened this issue · comments
Environment
- How did you install GRR? [e.g. from release DEB, HEAD DEB, PIP package, source]
PIP package/source - What GRR version are you running?: [e.g. 3.1.2.3]
3.4.5.1 - What operating system does the GRR server run on? [e.g. Ubuntu 18.04]
Debian 11 (Bullseye) (ppc64le) - What operating system does the affected GRR client run on, if applicable? [e.g. Windows 10]
Describe the issue
Trying to install GRR Server from pip packages or via source, I am not able to install fleetspeak-client-bin version 0.1.11 on a ppc64le architecture.
Steps to reproduce:
pip3 install fleetspeak-client-bin
Error logs
ERROR: Could not find a version that satisfies the requirement fleetspeak-client-bin
ERROR: No matching distribution found for fleetspeak-client-bin
Additional context
There is a working (old) version of GRR Server (ppc64el) at https://packages.ubuntu.com/bionic/grr-server for Ubuntu 18.04.
Quick question: do you plan to run the clients on ppc64le
as well?
fleetspeak-client-bin contains precompiled Fleetspeak binaries for various platforms and is needed to build and run latest generation of GRR clients. Unfortunately, we don't distribute the ppc64le Fleetspeak binaries in the fleetspeak-client-bin
package.
At the moment, in order to install GRR and then build client templates for ppc64le, you'd need to effectively build and install your own ppc64le-specific fleetspeak-client-bin package version. To do that, you'd essentially need to follow the steps done by Fleetspeak CI.
Note that even if you do that, you'd still need to build the custom GRR client template for pcc64le
too: https://grr-doc.readthedocs.io/en/latest/maintaining-and-tuning/building-custom-client-templates.html
Answer: I do plan to run the clients on ppc64le
as well.
I think I was able to follow the steps done by Fleetspeak CI and created fleetspeak-client_0.1.11_ppc64el.deb
and fleetspeak-server_0.1.11_ppc64el.deb
. I was able to install them on the server, however following https://grr-doc.readthedocs.io/en/latest/installing-grr-server/from-source.html#installing-python-3-grr-version-from-source I run into a few errors with dependencies (fleetspeak-client-bin and fleetspeak-server-bin) listed in several setup.py. I commented them out one by one and got further. I am now getting the following errors:
Building wheels for collected packages: mysqlclient
Building wheel for mysqlclient (setup.py) ... error
ERROR: Command errored out with exit status 1:
command: /path/xyz/INSTALL/bin/python3 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-igw5ajvu/mysqlclient_77193281ccd143ee9eca01f0d4efb271/setup.py'"'"'; __file__='"'"'/tmp/pip-install-igw5ajvu/mysqlclient_77193281ccd143ee9eca01f0d4efb271/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-x34f4rh2
cwd: /tmp/pip-install-igw5ajvu/mysqlclient_77193281ccd143ee9eca01f0d4efb271/
Complete output (81 lines):
/path/xyz/INSTALL/lib/python3.9/site-packages/setuptools/dist.py:717: UserWarning: Usage of dash-separated 'distribution-name' will not be supported in future versions. Please use the underscore name 'distribution_name' instead
warnings.warn(
/path/xyz/INSTALL/lib/python3.9/site-packages/setuptools/dist.py:717: UserWarning: Usage of dash-separated 'build-requires' will not be supported in future versions. Please use the underscore name 'build_requires' instead
warnings.warn(
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-ppc64le-3.9
copying _mysql_exceptions.py -> build/lib.linux-ppc64le-3.9
creating build/lib.linux-ppc64le-3.9/MySQLdb
copying MySQLdb/__init__.py -> build/lib.linux-ppc64le-3.9/MySQLdb
copying MySQLdb/compat.py -> build/lib.linux-ppc64le-3.9/MySQLdb
copying MySQLdb/connections.py -> build/lib.linux-ppc64le-3.9/MySQLdb
copying MySQLdb/converters.py -> build/lib.linux-ppc64le-3.9/MySQLdb
copying MySQLdb/cursors.py -> build/lib.linux-ppc64le-3.9/MySQLdb
copying MySQLdb/release.py -> build/lib.linux-ppc64le-3.9/MySQLdb
copying MySQLdb/times.py -> build/lib.linux-ppc64le-3.9/MySQLdb
creating build/lib.linux-ppc64le-3.9/MySQLdb/constants
copying MySQLdb/constants/__init__.py -> build/lib.linux-ppc64le-3.9/MySQLdb/constants
copying MySQLdb/constants/CLIENT.py -> build/lib.linux-ppc64le-3.9/MySQLdb/constants
copying MySQLdb/constants/CR.py -> build/lib.linux-ppc64le-3.9/MySQLdb/constants
copying MySQLdb/constants/ER.py -> build/lib.linux-ppc64le-3.9/MySQLdb/constants
copying MySQLdb/constants/FIELD_TYPE.py -> build/lib.linux-ppc64le-3.9/MySQLdb/constants
copying MySQLdb/constants/FLAG.py -> build/lib.linux-ppc64le-3.9/MySQLdb/constants
copying MySQLdb/constants/REFRESH.py -> build/lib.linux-ppc64le-3.9/MySQLdb/constants
running build_ext
building '_mysql' extension
creating build/temp.linux-ppc64le-3.9
powerpc64le-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -ffile-prefix-map=/build/python3.9-lkUtLV/python3.9-3.9.2=. -fstack-protector-strong -Wformat -Werror=format-security -fexceptions -g -fwrapv -O2 -g -ffile-prefix-map=/build/python3.9-lkUtLV/python3.9-3.9.2=. -fstack-protector-strong -Wformat -Werror=format-security -fexceptions -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Dversion_info=(1,3,10,'final',0) -D__version__=1.3.10 -I/usr/include/mariadb -I/usr/include/mariadb/mysql -I/path/xyz/INSTALL/include -I/usr/include/python3.9 -c _mysql.c -o build/temp.linux-ppc64le-3.9/_mysql.o
In file included from _mysql.c:29:
/usr/include/mariadb/my_config.h:3:2: warning: #warning This file should not be included by clients, include only <mysql.h> [-Wcpp]
3 | #warning This file should not be included by clients, include only <mysql.h>
| ^~~~~~~
_mysql.c: In function ‘_mysql_server_init’:
_mysql.c:266:6: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
266 | s = PyUnicode_AsUTF8(item);
| ^
_mysql.c:296:6: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
296 | s = PyUnicode_AsUTF8(item);
| ^
_mysql.c: In function ‘_mysql_ConnectionObject_Initialize’:
_mysql.c:577:16: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
577 | if(t){d=PyUnicode_AsUTF8(t);ssl_keepref[n_ssl_keepref++]=t;}\
| ^
_mysql.c:588:3: note: in expansion of macro ‘_stringsuck’
588 | _stringsuck(ca, value, ssl);
| ^~~~~~~~~~~
_mysql.c:577:16: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
577 | if(t){d=PyUnicode_AsUTF8(t);ssl_keepref[n_ssl_keepref++]=t;}\
| ^
_mysql.c:589:3: note: in expansion of macro ‘_stringsuck’
589 | _stringsuck(capath, value, ssl);
| ^~~~~~~~~~~
_mysql.c:577:16: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
577 | if(t){d=PyUnicode_AsUTF8(t);ssl_keepref[n_ssl_keepref++]=t;}\
| ^
_mysql.c:590:3: note: in expansion of macro ‘_stringsuck’
590 | _stringsuck(cert, value, ssl);
| ^~~~~~~~~~~
_mysql.c:577:16: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
577 | if(t){d=PyUnicode_AsUTF8(t);ssl_keepref[n_ssl_keepref++]=t;}\
| ^
_mysql.c:591:3: note: in expansion of macro ‘_stringsuck’
591 | _stringsuck(key, value, ssl);
| ^~~~~~~~~~~
_mysql.c:577:16: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
577 | if(t){d=PyUnicode_AsUTF8(t);ssl_keepref[n_ssl_keepref++]=t;}\
| ^
_mysql.c:592:3: note: in expansion of macro ‘_stringsuck’
592 | _stringsuck(cipher, value, ssl);
| ^~~~~~~~~~~
_mysql.c: In function ‘_mysql_ConnectionObject_ping’:
_mysql.c:1911:41: error: ‘MYSQL’ {aka ‘struct st_mysql’} has no member named ‘reconnect’
1911 | if ( reconnect != -1 ) self->connection.reconnect = reconnect;
| ^
_mysql.c: In function ‘_mysql_ConnectionObject_getattro’:
_mysql.c:2603:8: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
2603 | cname = PyUnicode_AsUTF8(name);
| ^
error: command '/usr/bin/powerpc64le-linux-gnu-gcc' failed with exit code 1
----------------------------------------
ERROR: Failed building wheel for mysqlclient
Running setup.py clean for mysqlclient
Failed to build mysqlclient
Installing collected packages: mysqlclient, grr-response-server
Attempting uninstall: mysqlclient
Found existing installation: mysqlclient 2.1.0
Uninstalling mysqlclient-2.1.0:
Successfully uninstalled mysqlclient-2.1.0
Running setup.py install for mysqlclient ... error
ERROR: Command errored out with exit status 1:
command: /path/xyz/INSTALL/bin/python3 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-igw5ajvu/mysqlclient_77193281ccd143ee9eca01f0d4efb271/setup.py'"'"'; __file__='"'"'/tmp/pip-install-igw5ajvu/mysqlclient_77193281ccd143ee9eca01f0d4efb271/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-uoyz__bc/install-record.txt --single-version-externally-managed --compile --install-headers /path/xyz/INSTALL/include/site/python3.9/mysqlclient
cwd: /tmp/pip-install-igw5ajvu/mysqlclient_77193281ccd143ee9eca01f0d4efb271/
Complete output (83 lines):
/path/xyz/INSTALL/lib/python3.9/site-packages/setuptools/dist.py:717: UserWarning: Usage of dash-separated 'distribution-name' will not be supported in future versions. Please use the underscore name 'distribution_name' instead
warnings.warn(
/path/xyz/INSTALL/lib/python3.9/site-packages/setuptools/dist.py:717: UserWarning: Usage of dash-separated 'build-requires' will not be supported in future versions. Please use the underscore name 'build_requires' instead
warnings.warn(
running install
/path/xyz/INSTALL/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
running build
running build_py
creating build
creating build/lib.linux-ppc64le-3.9
copying _mysql_exceptions.py -> build/lib.linux-ppc64le-3.9
creating build/lib.linux-ppc64le-3.9/MySQLdb
copying MySQLdb/__init__.py -> build/lib.linux-ppc64le-3.9/MySQLdb
copying MySQLdb/compat.py -> build/lib.linux-ppc64le-3.9/MySQLdb
copying MySQLdb/connections.py -> build/lib.linux-ppc64le-3.9/MySQLdb
copying MySQLdb/converters.py -> build/lib.linux-ppc64le-3.9/MySQLdb
copying MySQLdb/cursors.py -> build/lib.linux-ppc64le-3.9/MySQLdb
copying MySQLdb/release.py -> build/lib.linux-ppc64le-3.9/MySQLdb
copying MySQLdb/times.py -> build/lib.linux-ppc64le-3.9/MySQLdb
creating build/lib.linux-ppc64le-3.9/MySQLdb/constants
copying MySQLdb/constants/__init__.py -> build/lib.linux-ppc64le-3.9/MySQLdb/constants
copying MySQLdb/constants/CLIENT.py -> build/lib.linux-ppc64le-3.9/MySQLdb/constants
copying MySQLdb/constants/CR.py -> build/lib.linux-ppc64le-3.9/MySQLdb/constants
copying MySQLdb/constants/ER.py -> build/lib.linux-ppc64le-3.9/MySQLdb/constants
copying MySQLdb/constants/FIELD_TYPE.py -> build/lib.linux-ppc64le-3.9/MySQLdb/constants
copying MySQLdb/constants/FLAG.py -> build/lib.linux-ppc64le-3.9/MySQLdb/constants
copying MySQLdb/constants/REFRESH.py -> build/lib.linux-ppc64le-3.9/MySQLdb/constants
running build_ext
building '_mysql' extension
creating build/temp.linux-ppc64le-3.9
powerpc64le-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -ffile-prefix-map=/build/python3.9-lkUtLV/python3.9-3.9.2=. -fstack-protector-strong -Wformat -Werror=format-security -fexceptions -g -fwrapv -O2 -g -ffile-prefix-map=/build/python3.9-lkUtLV/python3.9-3.9.2=. -fstack-protector-strong -Wformat -Werror=format-security -fexceptions -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Dversion_info=(1,3,10,'final',0) -D__version__=1.3.10 -I/usr/include/mariadb -I/usr/include/mariadb/mysql -I/path/xyz/INSTALL/include -I/usr/include/python3.9 -c _mysql.c -o build/temp.linux-ppc64le-3.9/_mysql.o
In file included from _mysql.c:29:
/usr/include/mariadb/my_config.h:3:2: warning: #warning This file should not be included by clients, include only <mysql.h> [-Wcpp]
3 | #warning This file should not be included by clients, include only <mysql.h>
| ^~~~~~~
_mysql.c: In function ‘_mysql_server_init’:
_mysql.c:266:6: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
266 | s = PyUnicode_AsUTF8(item);
| ^
_mysql.c:296:6: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
296 | s = PyUnicode_AsUTF8(item);
| ^
_mysql.c: In function ‘_mysql_ConnectionObject_Initialize’:
_mysql.c:577:16: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
577 | if(t){d=PyUnicode_AsUTF8(t);ssl_keepref[n_ssl_keepref++]=t;}\
| ^
_mysql.c:588:3: note: in expansion of macro ‘_stringsuck’
588 | _stringsuck(ca, value, ssl);
| ^~~~~~~~~~~
_mysql.c:577:16: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
577 | if(t){d=PyUnicode_AsUTF8(t);ssl_keepref[n_ssl_keepref++]=t;}\
| ^
_mysql.c:589:3: note: in expansion of macro ‘_stringsuck’
589 | _stringsuck(capath, value, ssl);
| ^~~~~~~~~~~
_mysql.c:577:16: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
577 | if(t){d=PyUnicode_AsUTF8(t);ssl_keepref[n_ssl_keepref++]=t;}\
| ^
_mysql.c:590:3: note: in expansion of macro ‘_stringsuck’
590 | _stringsuck(cert, value, ssl);
| ^~~~~~~~~~~
_mysql.c:577:16: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
577 | if(t){d=PyUnicode_AsUTF8(t);ssl_keepref[n_ssl_keepref++]=t;}\
| ^
_mysql.c:591:3: note: in expansion of macro ‘_stringsuck’
591 | _stringsuck(key, value, ssl);
| ^~~~~~~~~~~
_mysql.c:577:16: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
577 | if(t){d=PyUnicode_AsUTF8(t);ssl_keepref[n_ssl_keepref++]=t;}\
| ^
_mysql.c:592:3: note: in expansion of macro ‘_stringsuck’
592 | _stringsuck(cipher, value, ssl);
| ^~~~~~~~~~~
_mysql.c: In function ‘_mysql_ConnectionObject_ping’:
_mysql.c:1911:41: error: ‘MYSQL’ {aka ‘struct st_mysql’} has no member named ‘reconnect’
1911 | if ( reconnect != -1 ) self->connection.reconnect = reconnect;
| ^
_mysql.c: In function ‘_mysql_ConnectionObject_getattro’:
_mysql.c:2603:8: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
2603 | cname = PyUnicode_AsUTF8(name);
| ^
error: command '/usr/bin/powerpc64le-linux-gnu-gcc' failed with exit code 1
----------------------------------------
Rolling back uninstall of mysqlclient
Moving to /path/xyz/INSTALL/lib/python3.9/site-packages/MySQLdb/
from /path/xyz/INSTALL/lib/python3.9/site-packages/~ySQLdb
Moving to /path/xyz/INSTALL/lib/python3.9/site-packages/mysqlclient-2.1.0.dist-info/
from /path/xyz/INSTALL/lib/python3.9/site-packages/~ysqlclient-2.1.0.dist-info
ERROR: Command errored out with exit status 1: /path/xyz/INSTALL/bin/python3 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-igw5ajvu/mysqlclient_77193281ccd143ee9eca01f0d4efb271/setup.py'"'"'; __file__='"'"'/tmp/pip-install-igw5ajvu/mysqlclient_77193281ccd143ee9eca01f0d4efb271/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-uoyz__bc/install-record.txt --single-version-externally-managed --compile --install-headers /path/xyz/INSTALL/include/site/python3.9/mysqlclient Check the logs for full command output.
Do I already need an existing MySQL/MariaDB setup at this point? (travis/install.sh
) Is there another missing dependency?
Is Ubuntu 18.04 still the recommended distribution to install the server? I would like to use a distribution and version still getting updates for the GRR Rapid Response setup.
Hey, our docs on the matter were a little outdated, I created a PR (google/grr-doc#144) to fix this.
Before the PR gets submitted, the temp version of the docs is available here: https://grr-doc.readthedocs.io/en/sphinx-update/installing-grr-server/from-source.html
Commenting on your questions/notes:
Answer: I do plan to run the clients on ppc64le as well.
Is it a hard requirement to run the server on the ppc64le architecture? That's not impossible but is much more work, as ppc64le
is not a tested scenario and we don't provide prebuilt server Debs for this architecture. A possible option might be: run the server on x86-64 Ubuntu 18 and build a custom client template for ppc64le
(building a custom client template doesn't involve installing mysql and building the server pip, thus making the process much simpler).
Do I already need an existing MySQL/MariaDB setup at this point? (travis/install.sh) Is there another missing dependency?
The error that you see in the build are not a sign of a missing dependency, but rather a build error inside the mysql-client
package. I found similar mentions of the same problem: DefectDojo/django-DefectDojo#407
You don't need to have an initialized database, but you need to have the headers/libraries (which you seem to have, otherwise the build would fail totally).
Which database did you install - MySQL or MariaDB?
Is Ubuntu 18.04 still the recommended distribution to install the server? I would like to use a distribution and version still getting updates for the GRR Rapid Response setup.
Yes, Ubuntu 18.04 is the server environment that we support and test on.