This project looks amazing! (I'm getting an error though...)
alexlenail opened this issue · comments
Hello!
I've been looking around for something just like this. I'm a beginner with travis so I hope you'll bear with me.
My code is here and the relevant log file can be found here. I can build wheels locally, upload to pypi and download and install, but my friend can't, so I'm hoping to build many-platform wheels. I think the issue has to do with travis' isolated environments
The command "pip install cibuildwheel==0.1.3 pybind11==2.1.0" exited with 0.
80.32s$ cibuildwheel --output-dir wheelhouse
+ curl -L -o /tmp/Python.pkg https://www.python.org/ftp/python/2.7.13/python-2.7.13-macosx10.6.pkg
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 21.4M 100 21.4M 0 0 2950k 0 0:00:07 0:00:07 --:--:-- 3594k
+ sudo installer -pkg /tmp/Python.pkg -target /
installer: Package name is Python
installer: Installing at base path /
installer: The install was successful.
+ which python2
/Library/Frameworks/Python.framework/Versions/2.7/bin/python2
+ python2 --version
Python 2.7.13
+ python2 -m ensurepip --upgrade
Requirement already up-to-date: setuptools in /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages
Requirement already up-to-date: pip in /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages
+ pip2 --version
pip 9.0.1 from /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages (python 2.7)
+ pip2 install wheel
Collecting wheel
Downloading wheel-0.29.0-py2.py3-none-any.whl (66kB)
100% |████████████████████████████████| 71kB 1.0MB/s
Installing collected packages: wheel
Successfully installed wheel-0.29.0
+ pip2 install delocate
Collecting delocate
Downloading delocate-0.6.4-py2-none-any.whl (85kB)
100% |████████████████████████████████| 92kB 2.5MB/s
Requirement already satisfied: wheel in /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages (from delocate)
Installing collected packages: delocate
Successfully installed delocate-0.6.4
+ pip2 wheel . -w /tmp/tmpwheel2.7
Processing /Users/travis/build/fraenkel-lab/pcst_fast
Collecting pybind11==2.1.0 (from pcst-fast==1.0.2)
Using cached pybind11-2.1.0-py2.py3-none-any.whl
Saved /private/tmp/tmpwheel2.7/pybind11-2.1.0-py2.py3-none-any.whl
Skipping pybind11, due to already being wheel.
Building wheels for collected packages: pcst-fast
Running setup.py bdist_wheel for pcst-fast ... error
Complete output from command /Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python -u -c "import setuptools, tokenize;__file__='/private/var/folders/my/m6ynh3bn6tq06h7xr3js0z7r0000gn/T/pip-vBbxQd-build/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /var/folders/my/m6ynh3bn6tq06h7xr3js0z7r0000gn/T/tmppPIxUUpip-wheel-:
running bdist_wheel
running build
running build_py
package init file 'src/__init__.py' not found (or not a regular file)
creating build
creating build/lib.macosx-10.6-intel-2.7
creating build/lib.macosx-10.6-intel-2.7/pcst_fast
copying src/test_pcst_fast.py -> build/lib.macosx-10.6-intel-2.7/pcst_fast
running build_ext
creating var
creating var/folders
creating var/folders/my
creating var/folders/my/m6ynh3bn6tq06h7xr3js0z7r0000gn
creating var/folders/my/m6ynh3bn6tq06h7xr3js0z7r0000gn/T
/usr/bin/clang -fno-strict-aliasing -fno-common -dynamic -arch i386 -arch x86_64 -g -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c /var/folders/my/m6ynh3bn6tq06h7xr3js0z7r0000gn/T/tmpgO7CCO.cpp -o var/folders/my/m6ynh3bn6tq06h7xr3js0z7r0000gn/T/tmpgO7CCO.o -std=c++14
/usr/bin/clang -fno-strict-aliasing -fno-common -dynamic -arch i386 -arch x86_64 -g -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c /var/folders/my/m6ynh3bn6tq06h7xr3js0z7r0000gn/T/tmpOnS03A.cpp -o var/folders/my/m6ynh3bn6tq06h7xr3js0z7r0000gn/T/tmpOnS03A.o -fvisibility=hidden
building 'pcst_fast' extension
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/private/var/folders/my/m6ynh3bn6tq06h7xr3js0z7r0000gn/T/pip-vBbxQd-build/setup.py", line 103, in <module>
zip_safe=False,
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 151, in setup
dist.run_commands()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/wheel/bdist_wheel.py", line 179, in run
self.run_command('build')
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 326, in run_command
self.distribution.run_command(command)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/build.py", line 127, in run
self.run_command(cmd_name)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 326, in run_command
self.distribution.run_command(command)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/setuptools/command/build_ext.py", line 75, in run
_build_ext.run(self)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/build_ext.py", line 340, in run
self.build_extensions()
File "/private/var/folders/my/m6ynh3bn6tq06h7xr3js0z7r0000gn/T/pip-vBbxQd-build/setup.py", line 88, in build_extensions
build_ext.build_extensions(self)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/build_ext.py", line 449, in build_extensions
self.build_extension(ext)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/setuptools/command/build_ext.py", line 196, in build_extension
_build_ext.build_extension(self, ext)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/build_ext.py", line 499, in build_extension
depends=ext.depends)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/ccompiler.py", line 566, in compile
depends, extra_postargs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/ccompiler.py", line 355, in _setup_compile
pp_opts = gen_preprocess_options(macros, incdirs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/ccompiler.py", line 1052, in gen_preprocess_options
pp_opts.append ("-I%s" % dir)
File "/private/var/folders/my/m6ynh3bn6tq06h7xr3js0z7r0000gn/T/pip-vBbxQd-build/setup.py", line 18, in __str__
import pybind11
ImportError: No module named pybind11
----------------------------------------
Failed building wheel for pcst-fast
It looks like it installs pybind11, then installs python, then installs pybind11 again? But then can't find pybind11 when it tries to make the wheel. What do you make of this?
Thanks!
Hey thanks for this!
Yeah it's a little confusing, but I think that pybind11 is needed for the build. Now, that is correctly listed as a dependency in your setup.py install_requires, but pip isn't preinstalling that before trying to build the wheel. 🤔
I note that there's a setup.py field called setup_requires - maybe try adding the pybind11 library to that?
btw, the reason that doing pip install pybind11 before cibuildwheel doesn't work is that they are installed into different python environments. The default python environment on Travis is a homebrew-supplied one I think, that lives at /usr/local/bin/python. Cibuildwheel uses MacPython distributions which are installed into /Library/Frameworks/Python.framework
@joerick thanks for the information! Adding setup_requires gets me a different error which I think comes later but is related. Before, After.
Before:
File "/private/var/folders/my/m6ynh3bn6tq06h7xr3js0z7r0000gn/T/pip-vBbxQd-build/setup.py", line 18, in __str__
import pybind11
ImportError: No module named pybind11
----------------------------------------
Failed building wheel for pcst-fast
After
gcc -pthread -fno-strict-aliasing -Wformat -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/opt/_internal/cpython-2.6.9-ucs2/include/python2.6 -I/root/.local/include/python2.6 -I/opt/_internal/cpython-2.6.9-ucs2/include/python2.6 -c src/pcst_fast_pybind.cc -o build/temp.linux-x86_64-2.6/src/pcst_fast_pybind.o -DVERSION_INFO="1.0.2" -std=c++11 -fvisibility=hidden
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ [enabled by default]
src/pcst_fast_pybind.cc:6:31: fatal error: pybind11/pybind11.h: No such file or directory
#include <pybind11/pybind11.h>
^
compilation terminated.
error: command 'gcc' failed with exit status 1
----------------------------------------
Failed building wheel for pcst-fast
Any ideas?
Could you post the whole 'after' output? I think gcc isn't being invoked with the right include paths... maybe pybind needs to be installed globally for that to work
Hi @joerick, here's the whole output:
The command "pip install cibuildwheel==0.1.3 pybind11==2.1.0" exited with 0.
48.13s$ cibuildwheel --output-dir wheelhouse
Unable to find image 'quay.io/pypa/manylinux1_x86_64:latest' locally
latest: Pulling from pypa/manylinux1_x86_64
Status: Downloaded newer image for quay.io/pypa/manylinux1_x86_64:latest
+ cd /project
+ for PYBIN in '/opt/python/*/bin'
+ /opt/python/cp26-cp26m/bin/pip wheel . -w /tmp/linux_wheels
DEPRECATION: Python 2.6 is no longer supported by the Python core team, please upgrade your Python. A future version of pip will drop support for Python 2.6
Processing /project
Collecting pybind11==2.1.0 (from pcst-fast==1.0.2)
/opt/_internal/cpython-2.6.9-ucs2/lib/python2.6/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
SNIMissingWarning
/opt/_internal/cpython-2.6.9-ucs2/lib/python2.6/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
Downloading pybind11-2.1.0-py2.py3-none-any.whl (109kB)
Saved /tmp/linux_wheels/pybind11-2.1.0-py2.py3-none-any.whl
Skipping pybind11, due to already being wheel.
Building wheels for collected packages: pcst-fast
Running setup.py bdist_wheel for pcst-fast: started
Running setup.py bdist_wheel for pcst-fast: finished with status 'error'
Complete output from command /opt/_internal/cpython-2.6.9-ucs2/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-9aHlBZ-build/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmpaFXl8bpip-wheel-:
running bdist_wheel
running build
running build_py
package init file 'src/__init__.py' not found (or not a regular file)
creating build
creating build/lib.linux-x86_64-2.6
creating build/lib.linux-x86_64-2.6/pcst_fast
copying src/test_pcst_fast.py -> build/lib.linux-x86_64-2.6/pcst_fast
running build_ext
creating tmp
gcc -pthread -fno-strict-aliasing -Wformat -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/opt/_internal/cpython-2.6.9-ucs2/include/python2.6 -c /tmp/tmplasQ5j.cpp -o tmp/tmplasQ5j.o -std=c++11
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ [enabled by default]
gcc -pthread -fno-strict-aliasing -Wformat -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/opt/_internal/cpython-2.6.9-ucs2/include/python2.6 -c /tmp/tmptKJK0N.cpp -o tmp/tmptKJK0N.o -fvisibility=hidden
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ [enabled by default]
building 'pcst_fast' extension
creating build/temp.linux-x86_64-2.6
creating build/temp.linux-x86_64-2.6/src
gcc -pthread -fno-strict-aliasing -Wformat -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/opt/_internal/cpython-2.6.9-ucs2/include/python2.6 -I/root/.local/include/python2.6 -I/opt/_internal/cpython-2.6.9-ucs2/include/python2.6 -c src/pcst_fast_pybind.cc -o build/temp.linux-x86_64-2.6/src/pcst_fast_pybind.o -DVERSION_INFO="1.0.2" -std=c++11 -fvisibility=hidden
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ [enabled by default]
src/pcst_fast_pybind.cc:6:31: fatal error: pybind11/pybind11.h: No such file or directory
#include <pybind11/pybind11.h>
^
compilation terminated.
error: command 'gcc' failed with exit status 1
----------------------------------------
Failed building wheel for pcst-fast
It seems to get one step further now. Before "import pybind11" failed. Now that's succeeding, but when it goes to creating build/temp.linux-x86_64-2.6/src it can't find pybind11.h which is called for at the top of my c++ file.
How would you install globally?
@joerick This is how the include is done in the demo version of setup.py provided with pybind11 (adapted slightly). Seems reasonable to me, but I'm not totally sure in what order all of these events occur. Kind of strange that I can build the wheel locally no problem...
class get_pybind_include(object):
"""Helper class to determine the pybind11 include path
The purpose of this class is to postpone importing pybind11
until it is actually installed, so that the ``get_include()``
method can be invoked. """
def __init__(self, user=False):
self.user = user
def __str__(self):
import pybind11
return pybind11.get_include(self.user)
ext_modules = [
Extension(
'pcst_fast',
sources=['src/pcst_fast_pybind.cc', 'src/pcst_fast.cc'],
include_dirs=[
# Path to pybind11 headers
get_pybind_include(),
get_pybind_include(user=True)
],
language='c++'
),
]
Interestingly, someone here seems to be having a similar issue: pybind/python_example#18
I think that when the project is listed in setup_requires it's installed into an egg by easy_install, so when your setup script asks pip where to find headers it returns the wrong path. Maybe.
I'm going to look at adding an option CIBW_BEFORE_BUILD. I'm thinking that if you put CIBW_BEFORE_BUILD=pip install pybind11 that'll solve your problem. We'll see!
Hi @joerick
Thanks for getting back to me! That sounds like a great option to add for this use case. Keep me posted on how it goes!
Hey. I've added that feature, it's available on a feature branch of this repo for now. You can install with pip install git+https://github.com/joerick/cibuildwheel@2c0d6e3db1590a9f6902c6d46a8ff27aeb4f1add.
Would be interested if this allows you to solve your problem!
@joerick One of two has run, and it didn't work strangely. Here's my travis file by the way. I'm still waiting for the second build. (oh, and here's my setup.py copied from here.
Hooray! It worked! Got a new error though, which seems to have to do with the compiler? We don't get those error messages when we compile locally...
The second build errored in a really interesting way because it seems like it was able to build the wheel but not install it. Strange...
Thanks for the help @joerick. What will you do with this new feature? Will you merge into master?
Ah yeah, on Mac you need to use pip3 instead of pip so that the package installs into the right environment.
I added {pip} for this purpose. If you change your option to {pip} install ... as the option that should change to pip or pip3 as required.
Btw, I wrote docs for the option here 2c0d6e3
@joerick Amazing! One of the travis builds succeeded! I'm not sure I understand how the two of them are split up? The other one seems to fail because it has the wrong compiler? Link to travis page.
Oh I see, it succeeded on mac but failed on linux.
The thunk?
@joerick So I would either need to include that .h file in my project or CIBW_SKIP 2.6? I'd much prefer the latter, but it sounds like CIBW_SKIP will be fairly tough to implement?
@joerick Looks like I'm building all the wheels now in linux with CIBW_SKIP - (thanks by the way!) but it seems like it's trying to repair the pybind11 wheel at the end, which is failing: see the log. Is there some way to skip that as well? Perhaps this might be a problem arising from CIBW_BEFORE_BUILD?
I think this is because pip wheel is outputting your dependency wheel as well as the one you want. I've added a commit to restrict auditwheel and delocate to the package we're building.
pip install git+https://github.com/joerick/cibuildwheel@d5fa5475d4e8c58000d056f44a9968f68337cf95
p.s. thank you for your patience and help with this! we're ironing out a lot of kinks in this project that will hopefully save a lot of debugging time for future users! 👍
Hi @joerick. No worries, I'm happy to go through this. From my perspective, you're one of the most responsive open source developers I've ever had the pleasure to work with! I think this project will be super helpful to me as well as many others, and I'm happy to help out ironing the kinks =)
Looks like your build failed again
https://travis-ci.org/fraenkel-lab/pcst_fast/jobs/221348812
I think I'm guessing the wheel name wrong 🤔
@joerick you're one step ahead of me.
It's worth noting that when you added CIBW_SKIP the mac builds started failing (whereas they had been working when you added CIBW_BEFORE.... I'm not entirely sure why. Check it out.
Turns out it was a weird bug related to setup_requires. Which is now fixed. Let me know how you get on!
pip install git+https://github.com/joerick/cibuildwheel@c3565d5025566dcf4b795eef8d40f38a12872151
@joerick Looks like it's all working now! Just need to figure out how to automagically put the wheels on pypi.
🙌 do you mind if I add your repo to the readme as an example?
Of course! No Problem! I think that particular repo of ours will be around for a long time.
@joerick interestingly, we're having a bug with python 3.5 / macOS (which is resolved in the python 3.6 wheel):
>>> import pcst_fast
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pcst_fast.cpython-35m-darwin.so, 2): Symbol not found: __PyThreadState_UncheckedGet
Referenced from: /Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pcst_fast.cpython-35m-darwin.so
Expected in: flat namespace
in /Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pcst_fast.cpython-35m-darwin.so
Seems like the relevant part is Symbol not found: __PyThreadState_UncheckedGet. Do you know anything about this Symbol? I'm not quite sure where to look for this bug...
Hmm, never seen that before. Anything suspicious in the build log for that wheel?
Not that I can tell. That's super weird... How is it possible that this successfully installs the wheel on travis but not on real Macs with python 3.5? Could this have to do with minor versions? (I think it fails on 3.5.1, the wheel was built for 3.5.3...) @joerick