error: sodium.h: No such file or directory
fletom opened this issue · comments
PyNaCl installs fine using pip, but when imported it bugs out saying it can't find a file called sodium.h.
From my understanding, PyNaCl includes libsodium and doesn't need it to be pre-installed on the system. I also tried pre-installing it just to make sure, and got the same error. I also verified that sodium.h was found in my LD_LIBRARY_PATH, but no matter what I do I seem to get the same error.
Has anybody ever seen this before?
If you would like to reproduce it the exact environment (Heroku) you can do so by running:
git init
echo "PyNaCl" > requirements.txt
echo "cffi" >> requirements.txt
heroku create --buildpack git://github.com/kennethjiang/heroku-buildpack-python-libffi.git
git add .
git commit -m "test"
git push heroku master
heroku run 'python -c "import nacl.hash"'
The full content of the error I get is as follows:
~ $ python -c "import nacl.hash"
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:185:20: error: sodium.h: No such file or directory
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_box’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:269: warning: implicit declaration of function ‘crypto_box’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_box_afternm’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:345: warning: implicit declaration of function ‘crypto_box_afternm’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_box_beforenm’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:402: warning: implicit declaration of function ‘crypto_box_beforenm’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_box_beforenmbytes’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:416: warning: implicit declaration of function ‘crypto_box_beforenmbytes’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_box_boxzerobytes’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:430: warning: implicit declaration of function ‘crypto_box_boxzerobytes’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_box_keypair’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:474: warning: implicit declaration of function ‘crypto_box_keypair’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_box_noncebytes’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:488: warning: implicit declaration of function ‘crypto_box_noncebytes’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_box_open’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:577: warning: implicit declaration of function ‘crypto_box_open’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_box_open_afternm’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:653: warning: implicit declaration of function ‘crypto_box_open_afternm’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_box_publickeybytes’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:667: warning: implicit declaration of function ‘crypto_box_publickeybytes’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_box_secretkeybytes’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:681: warning: implicit declaration of function ‘crypto_box_secretkeybytes’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_box_zerobytes’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:695: warning: implicit declaration of function ‘crypto_box_zerobytes’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_hash’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:745: warning: implicit declaration of function ‘crypto_hash’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_hash_sha256’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:795: warning: implicit declaration of function ‘crypto_hash_sha256’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_hash_sha256_bytes’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:809: warning: implicit declaration of function ‘crypto_hash_sha256_bytes’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_hash_sha512’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:859: warning: implicit declaration of function ‘crypto_hash_sha512’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_hash_sha512_bytes’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:873: warning: implicit declaration of function ‘crypto_hash_sha512_bytes’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_scalarmult_base’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:917: warning: implicit declaration of function ‘crypto_scalarmult_base’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_scalarmult_bytes’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:931: warning: implicit declaration of function ‘crypto_scalarmult_bytes’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_scalarmult_scalarbytes’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:945: warning: implicit declaration of function ‘crypto_scalarmult_scalarbytes’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_secretbox’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:1021: warning: implicit declaration of function ‘crypto_secretbox’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_secretbox_boxzerobytes’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:1035: warning: implicit declaration of function ‘crypto_secretbox_boxzerobytes’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_secretbox_keybytes’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:1049: warning: implicit declaration of function ‘crypto_secretbox_keybytes’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_secretbox_noncebytes’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:1063: warning: implicit declaration of function ‘crypto_secretbox_noncebytes’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_secretbox_open’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:1139: warning: implicit declaration of function ‘crypto_secretbox_open’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_secretbox_zerobytes’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:1153: warning: implicit declaration of function ‘crypto_secretbox_zerobytes’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_sign’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:1229: warning: implicit declaration of function ‘crypto_sign’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_sign_bytes’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:1243: warning: implicit declaration of function ‘crypto_sign_bytes’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_sign_keypair’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:1287: warning: implicit declaration of function ‘crypto_sign_keypair’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_sign_open’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:1363: warning: implicit declaration of function ‘crypto_sign_open’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_sign_publickeybytes’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:1377: warning: implicit declaration of function ‘crypto_sign_publickeybytes’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_sign_secretkeybytes’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:1391: warning: implicit declaration of function ‘crypto_sign_secretkeybytes’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_crypto_sign_seed_keypair’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:1448: warning: implicit declaration of function ‘crypto_sign_seed_keypair’
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c: In function ‘_cffi_f_randombytes’:
.heroku/python/lib/python2.7/site-packages/nacl/_lib/__pycache__/_cffi__xac3ef8b2x44e2af4a.c:1484: warning: implicit declaration of function ‘randombytes’
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/app/.heroku/python/lib/python2.7/site-packages/nacl/hash.py", line 17, in <module>
import nacl.c
File "/app/.heroku/python/lib/python2.7/site-packages/nacl/c/__init__.py", line 16, in <module>
from nacl.c.crypto_box import (
File "/app/.heroku/python/lib/python2.7/site-packages/nacl/c/crypto_box.py", line 23, in <module>
crypto_box_SECRETKEYBYTES = lib.crypto_box_secretkeybytes()
File "/app/.heroku/python/lib/python2.7/site-packages/nacl/_lib/__init__.py", line 73, in __getattr__
self._lib = self.ffi.verifier.load_library()
File "/app/.heroku/python/lib/python2.7/site-packages/cffi/verifier.py", line 74, in load_library
self._compile_module()
File "/app/.heroku/python/lib/python2.7/site-packages/cffi/verifier.py", line 139, in _compile_module
outputfilename = ffiplatform.compile(tmpdir, self.get_extension())
File "/app/.heroku/python/lib/python2.7/site-packages/cffi/ffiplatform.py", line 25, in compile
outputfilename = _build(tmpdir, ext)
File "/app/.heroku/python/lib/python2.7/site-packages/cffi/ffiplatform.py", line 51, in _build
raise VerificationError('%s: %s' % (e.__class__.__name__, e))
cffi.ffiplatform.VerificationError: CompileError: command 'gcc' failed with exit status 1
~ $
Thanks!
I'm seeing this same behavior when installing pynacl via buildout. What's interesting is that I cannot reproduce it by creating a new virtualenv (either pypy 2.3.0 or cpython 2.7). My only conjecture is that in the OP's environment (and under buildout), the cffi bindings are not getting generated at install time. When attempting to build them at run time, it fails because libsodium is no longer available. I would appreciate being able to point pynacl at a different libsodium anyway than the bundled one. This is possible with other cffi-using projects (pyzmq springs to mind). From an audit perspective, it would be much easier to ensure that the library is legitimate independently.
I am testing on OSX 10.9.3 using MacPorts.
@davidblewett I'm not sure about why this is failing (and I don't have time right now to hunt down the problem), but you can tell pynacl to use the system's libsodium instead of the bundled one by setting a environmental variable. https://github.com/pyca/pynacl/blob/master/setup.py#L80
I don't know if this is relevant, but if I run export SODIUM_INSTALL=system (when libsodium is installed system-wide) before running I still get this same error. Or does this variable have to be set during installation?
I really appreciate both of you looking at this. If there was some way I could get this working I'd be so grateful.
@fletom I was able to get this working with a non-bundled libsodium by passing C_INCLUDE_PATH and LIBRARY_PATH environment variables pointing to the correct locations the first time I imported an nacl module that uses a C interface (I chose nacl.signing). For an OS-supplied libsodium on Linux, those would probably be /usr/include and /usr/lib respectively. The first time nacl needs to access a C library, cffi will build the shared object, so you won't need to pass the environment variables explicitly after that.
What I'm not sure on is why I have to pass them at all the first time. It seems that the cffi shared object should be built at install time, using the environment passed then. When I install the package using buildout, I have the above 2 variables set so it should be able to do so.
@davidblewett That worked!
I had to set all four of the following:
export SODIUM_INSTALL=system
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:[libsodium's libs dir, where libsodium.a/.la/.so are found]
export LIBRARY_PATH=$LIBRARY_PATH:[libsodium's libs dir, same as above]
export C_INCLUDE_PATH=$C_INCLUDE_PATH:[libsodium's include dir, where sodium.h is found]
Each of these environment variables seems to be strictly necessary, as leaving out even one of them before importing something like nacl.hash causes some kind of compilation-related error to appear.
Anyways, if this gets fixed someday that's great, but for now I'm happy with having this workaround. Hopefully other people with the same problem will see this.
Thank you @davidblewett!
Compilation on import itself seem to be bogus - don't think cffi should do that, and it usually can't do that with module installed system-wide (e.g. in /usr), unless you're importing it as root.
I've had the same issue installing PyNaCl from PKGBUILD (having simple python2 setup.py install --root="$pkgdir/" --optimize=1 inside) on current ARMv7 Arch, but only when I forgot to include python2-cffi in "depends=(...)" there, so it wasn't available during build.
Building with python2-cffi installed produced proper pre-built package that imports without any compilation (as far as I can tell).
Not sure if build working without cffi like that is supposed to be a bug or a feature.
@mk-fg Re compilation, I agree. If it's possible not to compile anything at all on import that would be ideal. I notice that PyNaCl takes a few seconds to import (longer than any other packages) because of this.
I'm also seeing this error now on my local machine, when I have 'pynacl' in my setup.py's install_requires and I run python setup.py develop. Strangely, if I run pip install pynacl instead, I don't get the error. Is there something I need to add to my setup.py to get it working perhaps?
I'm actually seeing this same error when trying to install from the git repo using pip install --editable. The difference seems to be the presence of the --editable flag. I'm not sure if this is a separate issue, or if heroku create adds that flag somewhere.
If you have Docker installed, you can reproduce the error using:
docker build - <<'EOF'
FROM ubuntu:12.04
RUN apt-get -qy update
RUN apt-get -qy install build-essential git libffi-dev python-dev python-pip
# Uncomment one of the following 4 cases:
# 1. This fails:
RUN git clone https://github.com/pyca/pynacl && cd pynacl && pip install -e .
# 2. This also fails:
#RUN pip install -e 'git+https://github.com/pyca/pynacl@master#egg=pynacl'
# 3. This succeeds:
#RUN git clone https://github.com/pyca/pynacl && cd pynacl && pip install .
# 4. This also succeeds:
#RUN pip install 'git+https://github.com/pyca/pynacl@master#egg=pynacl'
EOF
This is with pynacl v0.2.3-97-g337a955.
As I'm bumping into this one again, and there doesn't seem to be a link to a related cffi packaging issue, here it is: https://bitbucket.org/cffi/cffi/issue/109/enable-sane-packaging-for-cffi
With the build changes that went into the recent 0.3.0 release, is this still a problem? I know that 0.3.0 fixed bug I was experiencing, in which a different package that had "install_requires=pynacl" could not be pip-installed. Could you check to see if anything's improved in the new version?
Sorry for very delayed response, but latest releases indeed don't seem to have this issue for me - packaging pre-built module works fine and doesn't trigger rebuild on first use.
great, thanks, I'll close this out.