Cannot pip install energyflow on M1 mac
mattleblanc opened this issue · comments
Hi,
I am unable to pip install energyflow on an M1 mac ... the resulting error messages are below.
I think there needs to be a cibuildwheel for M1 installations? @matthewfeickert might know better ...
🍻 MLB
(emd) eduroam-hci-dock-1-305:benchmarks mleblanc$ pip install energyflow
Collecting energyflow
Using cached EnergyFlow-1.3.2-py2.py3-none-any.whl (700 kB)
Collecting numpy>=1.16.0
Downloading numpy-1.23.0-cp39-cp39-macosx_11_0_arm64.whl (13.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.3/13.3 MB 16.4 MB/s eta 0:00:00
Collecting wasserstein>=0.3.1
Using cached Wasserstein-1.0.1.tar.gz (382 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Collecting six>=1.10.0
Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting h5py>=2.9.0
Using cached h5py-3.7.0-cp39-cp39-macosx_11_0_arm64.whl (2.6 MB)
Collecting wurlitzer>=2.0.0
Using cached wurlitzer-3.0.2-py3-none-any.whl (7.3 kB)
Building wheels for collected packages: wasserstein
Building wheel for wasserstein (pyproject.toml) ... error
error: subprocess-exited-with-error
× Building wheel for wasserstein (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [29 lines of output]
/private/var/folders/0c/bns72rts4694w4ftngm2_mh80000gn/T/pip-build-env-pqlly00e/overlay/lib/python3.9/site-packages/setuptools/config/setupcfg.py:463: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead.
warnings.warn(msg, warning_class)
running bdist_wheel
running build
running build_py
creating build
creating build/lib.macosx-12-arm64-cpython-39
creating build/lib.macosx-12-arm64-cpython-39/wasserstein
copying wasserstein/__init__.py -> build/lib.macosx-12-arm64-cpython-39/wasserstein
copying wasserstein/wasserstein.py -> build/lib.macosx-12-arm64-cpython-39/wasserstein
running build_ext
building 'wasserstein._wasserstein' extension
creating build/temp.macosx-12-arm64-cpython-39
creating build/temp.macosx-12-arm64-cpython-39/wasserstein
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -DSWIG_TYPE_TABLE=wasserstein -I/private/var/folders/0c/bns72rts4694w4ftngm2_mh80000gn/T/pip-build-env-pqlly00e/overlay/lib/python3.9/site-packages/numpy/core/include -I. -I/Users/mleblanc/svjets/benchmarks/emd/include -I/opt/homebrew/Cellar/python@3.9/3.9.12_1/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c wasserstein/wasserstein.cpp -o build/temp.macosx-12-arm64-cpython-39/wasserstein/wasserstein.o -Xpreprocessor -fopenmp -ffast-math -std=c++14 -g0
In file included from wasserstein/wasserstein.cpp:5457:
In file included from ./wasserstein/Wasserstein.hh:47:
In file included from wasserstein/internal/PairwiseEMD.hh:54:
wasserstein/internal/PairwiseEMDBase.hh:54:10: fatal error: 'omp.h' file not found
#include <omp.h>
^~~~~~~
clang: error: unable to execute command: Segmentation fault: 11
clang: error: clang frontend command failed due to signal (use -v to see invocation)
Apple clang version 13.1.6 (clang-1316.0.21.2.3)
Target: arm64-apple-darwin21.1.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
clang: note: diagnostic msg: Error generating preprocessed source(s).
error: command '/usr/bin/clang' failed with exit code 254
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for wasserstein
Failed to build wasserstein
ERROR: Could not build wheels for wasserstein, which is required to install pyproject.toml-based projects
(emd) eduroam-hci-dock-1-305:benchmarks mleblanc$ pip install wheel
Collecting wheel
Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB)
Installing collected packages: wheel
Successfully installed wheel-0.37.1
(emd) eduroam-hci-dock-1-305:benchmarks mleblanc$ pip install energyflow
Collecting energyflow
Using cached EnergyFlow-1.3.2-py2.py3-none-any.whl (700 kB)
Collecting numpy>=1.16.0
Using cached numpy-1.23.0-cp39-cp39-macosx_11_0_arm64.whl (13.3 MB)
Collecting h5py>=2.9.0
Using cached h5py-3.7.0-cp39-cp39-macosx_11_0_arm64.whl (2.6 MB)
Collecting wasserstein>=0.3.1
Using cached Wasserstein-1.0.1.tar.gz (382 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Collecting six>=1.10.0
Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting wurlitzer>=2.0.0
Using cached wurlitzer-3.0.2-py3-none-any.whl (7.3 kB)
Building wheels for collected packages: wasserstein
Building wheel for wasserstein (pyproject.toml) ... error
error: subprocess-exited-with-error
× Building wheel for wasserstein (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [29 lines of output]
/private/var/folders/0c/bns72rts4694w4ftngm2_mh80000gn/T/pip-build-env-cdkgzfz2/overlay/lib/python3.9/site-packages/setuptools/config/setupcfg.py:463: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead.
warnings.warn(msg, warning_class)
running bdist_wheel
running build
running build_py
creating build
creating build/lib.macosx-12-arm64-cpython-39
creating build/lib.macosx-12-arm64-cpython-39/wasserstein
copying wasserstein/__init__.py -> build/lib.macosx-12-arm64-cpython-39/wasserstein
copying wasserstein/wasserstein.py -> build/lib.macosx-12-arm64-cpython-39/wasserstein
running build_ext
building 'wasserstein._wasserstein' extension
creating build/temp.macosx-12-arm64-cpython-39
creating build/temp.macosx-12-arm64-cpython-39/wasserstein
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -DSWIG_TYPE_TABLE=wasserstein -I/private/var/folders/0c/bns72rts4694w4ftngm2_mh80000gn/T/pip-build-env-cdkgzfz2/overlay/lib/python3.9/site-packages/numpy/core/include -I. -I/Users/mleblanc/svjets/benchmarks/emd/include -I/opt/homebrew/Cellar/python@3.9/3.9.12_1/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c wasserstein/wasserstein.cpp -o build/temp.macosx-12-arm64-cpython-39/wasserstein/wasserstein.o -Xpreprocessor -fopenmp -ffast-math -std=c++14 -g0
In file included from wasserstein/wasserstein.cpp:5457:
In file included from ./wasserstein/Wasserstein.hh:47:
In file included from wasserstein/internal/PairwiseEMD.hh:54:
wasserstein/internal/PairwiseEMDBase.hh:54:10: fatal error: 'omp.h' file not found
#include <omp.h>
^~~~~~~
clang: error: unable to execute command: Segmentation fault: 11
clang: error: clang frontend command failed due to signal (use -v to see invocation)
Apple clang version 13.1.6 (clang-1316.0.21.2.3)
Target: arm64-apple-darwin21.1.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
clang: note: diagnostic msg: Error generating preprocessed source(s).
error: command '/usr/bin/clang' failed with exit code 254
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for wasserstein
Failed to build wasserstein
ERROR: Could not build wheels for wasserstein, which is required to install pyproject.toml-based projects
Yup, need a build for M1's custom ARM arch. I think Patrick is already familiar with how to setup new cibuildwheel arch builds, but check out awkward and boost-histogram for very well maintained examples (also of course the cibuildwheel project itself). (Sorry on phone else would link and give more description).
Though @mattleblanc you should also be able to build from source if you clone the repo and ensure you have all build dependencies. It should work for the sdist too (which is what pip is trying to build from on PyPI) but I haven't inspected the sdiat packaging.
Got it, thanks for pointing this out. EnergyFlow is pure python, but Wasserstein is not and it seems like the issue is originating there. Let me see if I can quickly compile that for macOS ARM.
Hi, should this work now or do you need to do something else? I still see it failing when I try.
I'm still working on it. Hopefully I will be done in a few hours.
I have released a new version of the Wasserstein package that includes wheels for mac arm64 chips. Run pip3 install -U wasserstein and then try to install EnergyFlow and it should succeed.
Hi @pkomiske, thanks for preparing this! I can install, but now I see strange errors when I try to import energyflow:
(emd) mleblanc-mbair:benchmarks mleblanc$ python
Python 3.9.12 (main, May 8 2022, 17:57:49)
[Clang 13.1.6 (clang-1316.0.21.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import energyflow
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/mleblanc/svjets/benchmarks/emd/lib/python3.9/site-packages/energyflow/__init__.py", line 39, in <module>
from . import emd
File "/Users/mleblanc/svjets/benchmarks/emd/lib/python3.9/site-packages/energyflow/emd.py", line 109, in <module>
_EMD = wasserstein.EMD()
File "/Users/mleblanc/svjets/benchmarks/emd/lib/python3.9/site-packages/wasserstein/__init__.py", line 93, in __getattr__
from . import wasserstein
File "/Users/mleblanc/svjets/benchmarks/emd/lib/python3.9/site-packages/wasserstein/__init__.py", line 101, in __getattr__
wasserstein = importlib.import_module('.wasserstein', __name__)
File "/opt/homebrew/Cellar/python@3.9/3.9.12_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/Users/mleblanc/svjets/benchmarks/emd/lib/python3.9/site-packages/wasserstein/wasserstein.py", line 19, in <module>
from . import _wasserstein
File "/Users/mleblanc/svjets/benchmarks/emd/lib/python3.9/site-packages/wasserstein/_wasserstein.py", line 31, in <module>
from ._wasserstein_omp import *
ImportError: dlopen(/Users/mleblanc/svjets/benchmarks/emd/lib/python3.9/site-packages/wasserstein/_wasserstein_omp.cpython-39-darwin.so, 0x0002): symbol not found in flat namespace '___kmpc_barrier'
It looks like some error involving openmp. Unfortunately I have no way to debug things for arm64 macs myself. Can you run the following:
import wasserstein
wasserstein.without_openmp()
import energyflow
This works! Happy to be a guinea pig, but I can try to go this way for local tests at the moment.
Does this completely disable multithreading, or does it just not use openmp?
This disables multithreading entirely, unfortunately. I introduced it as a not-ideal-but-all-I-could-think-of solution to the openmp library clashing with the one shipped with pytorch, so that at least there was a way to import the two packages together and not have it segfault.
Do you have libomp installed via homebrew, by chance? If so, there may be a way for now to use that with the package.
I have it installed by homebrew: how do I point energyflow at that?
I seem to be having similar problems with trying to install PyFJCore, for the record ...
Yeah PyFJCore will also need the mac arm64 wheels to be built. I'll get to that as soon as I can.
Try the following for the openmp issue: rm -rf /Users/mleblanc/svjets/benchmarks/emd/lib/python3.9/site-packages/wasserstein/.dylibs. This will delete the libomp.dylib that comes with wasserstein so then the system one provided by homebrew should hopefully be searched for and found (I confirmed that this works on my intel mac).
I don't seem to have any .dylibs in my copy of wasserstein --
(emd) mleblanc-mbair:benchmarks mleblanc$ ls -alh /Users/mleblanc/svjets/benchmarks/emd/lib/python3.9/site-packages/wasserstein/*
-rw-r--r-- 1 mleblanc staff 4.3K Jul 12 17:28 /Users/mleblanc/svjets/benchmarks/emd/lib/python3.9/site-packages/wasserstein/__init__.py
-rw-r--r-- 1 mleblanc staff 1.7K Jul 12 17:28 /Users/mleblanc/svjets/benchmarks/emd/lib/python3.9/site-packages/wasserstein/_wasserstein.py
-rwxr-xr-x 1 mleblanc staff 1.2M Jul 12 17:28 /Users/mleblanc/svjets/benchmarks/emd/lib/python3.9/site-packages/wasserstein/_wasserstein_noomp.cpython-39-darwin.so
-rwxr-xr-x 1 mleblanc staff 1.2M Jul 12 17:28 /Users/mleblanc/svjets/benchmarks/emd/lib/python3.9/site-packages/wasserstein/_wasserstein_omp.cpython-39-darwin.so
-rw-r--r-- 1 mleblanc staff 2.1K Jul 12 17:28 /Users/mleblanc/svjets/benchmarks/emd/lib/python3.9/site-packages/wasserstein/config.py
-rw-r--r-- 1 mleblanc staff 65K Jul 12 17:28 /Users/mleblanc/svjets/benchmarks/emd/lib/python3.9/site-packages/wasserstein/wasserstein.py
/Users/mleblanc/svjets/benchmarks/emd/lib/python3.9/site-packages/wasserstein/__pycache__:
total 136
drwxr-xr-x 6 mleblanc staff 192B Jul 12 17:28 .
drwxr-xr-x 9 mleblanc staff 288B Jul 12 17:28 ..
-rw-r--r-- 1 mleblanc staff 3.8K Jul 12 17:28 __init__.cpython-39.pyc
-rw-r--r-- 1 mleblanc staff 463B Jul 12 17:28 _wasserstein.cpython-39.pyc
-rw-r--r-- 1 mleblanc staff 914B Jul 12 17:28 config.cpython-39.pyc
-rw-r--r-- 1 mleblanc staff 54K Jul 12 17:28 wasserstein.cpython-39.pyc
Hi all, just coming through some old issues -- pip install energyflow works on ARM now, so this can probably be closed.