Failure to serialize in joblib
NikhilSehgal123 opened this issue · comments
Describe the bug
Consistently getting the following error when installing the faiss-cpu==1.7.3 module on Mac OS (Silicon M2 Chip):
ModuleNotFoundError: No module named 'faiss.swigfaiss_avx2'
To Reproduce
pip install faiss-cpu==1.7.3
Expected behavior
When installing the package, it is expected that the package scripts should include a script called:
swigfaiss_avx2.py
But this is missing when trying to install the module on my mac and I think this is causing the issue
Desktop (please complete the following information):
- OS: Mac
- Architecture: arm64
- Python: 3.9.16
- Version: v1.7.3
Additional context
This happens when trying to "joblib.load()"
There are no avx2 CPU instructions in arm64 arch.
@kyamagu does this mean it is not possible right now to use this lib on arm64 computers?
Binary wheels are available.
https://pypi.org/project/faiss-cpu/#files
Please post the exact crash log if anything is not working except showing ignorable error messages.
@kyamagu I tried a bunch of wheels but still getting an error:
pip install faiss_cpu-1.7.3-cp311-cp311-macosx_11_0_arm64.whl
ERROR: faiss_cpu-1.7.3-cp311-cp311-macosx_11_0_arm64.whl is not a supported wheel on this platform.
(env) (base) admin@Nikhils-MBP Downloads % pip install faiss_cpu-1.7.3-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
ERROR: faiss_cpu-1.7.3-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl is not a supported wheel on this platform.
(env) (base) admin@Nikhils-MBP Downloads % pip install faiss_cpu-1.7.3-cp310-cp310-macosx_11_0_arm64.whl
ERROR: faiss_cpu-1.7.3-cp310-cp310-macosx_11_0_arm64.whl is not a supported wheel on this platform.
(env) (base) admin@Nikhils-MBP Downloads % pip install faiss_cpu-1.7.3-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
ERROR: faiss_cpu-1.7.3-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl is not a supported wheel on this platform.
Just a quick check. Are you using x86_64 CPython or arm64 CPython on Apple Silicon?
@kyamagu using arm64 CPython on Apple Silicon
Ok, this will need to investigate why the arch match is not working. Add verbose flag
ERROR: faiss_cpu-1.7.3-cp311-cp311-macosx_11_0_arm64.whl is not a supported wheel on this platform.
ok thanks @kyamagu, let me know
@NikhilSehgal123 You've reported your env is the following
OS: Mac
Architecture: arm64
Python: 3.9.16
Version: v1.7.3
You have to use faiss_cpu-1.7.3-cp39-cp39-macosx_11_0_arm64.whl. Not a cp311 wheel.
@kyamagu this is now installing correctly, but I'm still getting the error:
ModuleNotFoundError: No module named 'faiss.swigfaiss_avx2'
When inspecting the package lib, I still see the script "swigfaiss_avx2.py" is missing.
On my other Intel Mac, I can see it successfully installs this script
Is the error message causing a crash? Or, is the program only reporting the error message? Post your log.
@kyamagu my log below:
(env) (base) admin@Nikhils-MBP neo-agent % /Users/admin/Desktop/Repos/ChatMSP/neo-agent/env/bin/python /Users/admin/Desktop/Repos/ChatMSP/neo-agent/neo_agent.py
Traceback (most recent call last):
File "/Users/admin/Desktop/Repos/ChatMSP/neo-agent/neo_agent.py", line 32, in
vectorscore_woodland_trust = joblib.load('runbooks/vectorstore_woodland_trust.pkl')
File "/Users/admin/Desktop/Repos/ChatMSP/neo-agent/env/lib/python3.9/site-packages/joblib/numpy_pickle.py", line 605, in load
obj = _unpickle(fobj, filename, mmap_mode)
File "/Users/admin/Desktop/Repos/ChatMSP/neo-agent/env/lib/python3.9/site-packages/joblib/numpy_pickle.py", line 529, in _unpickle
obj = unpickler.load()
File "/Users/admin/micromamba/lib/python3.9/pickle.py", line 1212, in load
dispatchkey[0]
File "/Users/admin/micromamba/lib/python3.9/pickle.py", line 1537, in load_stack_global
self.append(self.find_class(module, name))
File "/Users/admin/micromamba/lib/python3.9/pickle.py", line 1579, in find_class
import(module, level=0)
ModuleNotFoundError: No module named 'faiss.swigfaiss_avx2'
Looks like the "swigfaiss_avx2.py" script is not included in the package. If i check another PC, it works
I do not know what joblib is doing underneath, but I do not think any faiss component is picklable. If you are including a faiss index in the pickle, use faiss.write_index(). facebookresearch/faiss#421
@kyamagu that is strange because my script is running on virtual machines which seem to be ok. I will look into this further
Hi @kyamagu when running my script on a i386 arch, it works perfectly. The main difference I'm noticing is what happens when I run
pip install faiss-cpu
On the i386 arch, I see all the python scripts installed including a script called swigfaiss_avx2.py. However on the arm64 arch, this script is missing when you run the pip install.
Can you confirm if this is expected? When you run the pip install, do you also see a script called swigfaiss_avx2.py?
avx2 module is not available in arm64 wheel, because it is not possible to compile.
@kyamagu understood, thank you. I'll close this thread with the understanding this is not possible on arm64 arch