Possible incompatibility with macOS

Question

Possible incompatibility with macOS

AndreFCruz opened this issue 2 years ago · comments

Description

When importing fairgbm on a mac I immediately get the following error message:

OSError: dlopen(<env-dir>/python3.9/site-packages/fairgbm/lib_lightgbm.so, 0x0006): tried: '<env-dir>/python3.9/site-packages/fairgbm/lib_lightgbm.so' (not a mach-o file)

Screenshot:

I have tried this with Python 3.8, 3.9, and 3.10 (I haven't tried it with earlier Python versions due to incompatibility with arm64 CPUs, but I have no reason to believe the bug would be exclusive to this architecture).

Reproducible example

Install fairgbm on a mac
Run import fairgbm
See error message.

Environment info

fairgbm==0.9.13
macOS
arm64 (M1 Max)

Additional Comments

It also does not work when using a linux python3.9 environment via docker containers (on macOS).
With this setup, the following (different but related to the same file) error is shown:

OSError: /usr/local/lib/python3.9/site-packages/fairgbm/lib_lightgbm.so: cannot open shared object file: No such file or directory

João Veiga · Answer 1 · Fri Oct 28 2022 01:56:48 GMT+0800 (China Standard Time)

Hey @AndreFCruz :

Does it work on non-M1 macs?
Can you import lightgbm with the same version being used here in your M1?

Cheers 😊

Btw you mention Python 3.9 and above, but the screenshot is showing Python 3.8

André Cruz · Answer 2 · Fri Oct 28 2022 16:14:16 GMT+0800 (China Standard Time)

Thanks for the reply Veiga 😄

I just tested it and it does not work on my older Intel mac. It crashes with the same exact error:

The above screenshot is on a clean Python 3.7 environment, which should be the most stable version (but with 3.9 we get the same error on Intel as well).

Good point, I just tried it and I can't even pip install lightgbm on M1 (neither the older 3.2.1 nor the current version), but I can pip install fairgbm (but it crashes later on import).

However, the exception thrown when doing pip install lightgbm on M1 is different than the bug reported above, and I think it's a lightgbm bug. The general incompatibility with macs doesn't seem to be related to this one M1 thing, as I can install lightgbm fine on my Intel mac.

My current theory is that the Microsoft LightGBM setup builds the .so library for the current platform, and our pip package somehow has a pre-compiled library for linux only (fairgbm also installs much much faster than lightgbm). The exception is macOS asking for the mac-equivalent mach-o file.

The exception thrown when running pip install lightgbm on M1 is different and this one is likely due to the arm architecture:

Assertion failed: (_file->_atomsArrayCount == computedAtomCount && "more atoms allocated than expected")

(but one bug at a time; the first one definitely affects all macOS platforms).

PS: w.r.t. the first comment: I tested it on 3.8 and above, the repeated 3.9 was a typo.

André Cruz · Answer 3 · Tue Nov 22 2022 21:56:57 GMT+0800 (China Standard Time)

In the meantime I commented the tags for compatibility with MacOS and Windows from PyPI and linked this issue in the respective code block.

Plus added a line to the readme that currently only Linux compatibility is maintained.

Tânia Carvalho · Answer 4 · Wed Feb 08 2023 03:22:08 GMT+0800 (China Standard Time)

Hi @AndreFCruz ! I'm having the same issue :/ have you solved it in the meantime?

André Cruz · Answer 5 · Wed Feb 08 2023 17:29:26 GMT+0800 (China Standard Time)

Hi @tmcarvalho

Are you able to install and use the standard lightgbm package? (should be possible on intel-based macs)

If so, then you can install fairgbm directly from source instead of using pip install. Instructions are the same as with lightgbm (but using this repository as the source): https://lightgbm.readthedocs.io/en/latest/Installation-Guide.html#build-from-github

If you're using an arm mac, then (at least in my case) lightgbm can't be installed, which means fairgbm can't be installed either unfortunately.

Another option is getting access to a linux machine, as pip install fairgbm will work fine then.

Yijun Bian · Answer 6 · Thu Sep 21 2023 14:07:36 GMT+0800 (China Standard Time)

I encounter the same issue, and weirdly, lightgbm works well but fairgbm does not. Here is my error info

$ python
Python 3.8.17 | packaged by conda-forge | (default, Jun 16 2023, 07:11:32)
[Clang 14.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import lightgbm
>>> import fairgbm
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/elise/Software/miniconda3/envs/ensem/lib/python3.8/site-packages/fairgbm/__init__.py", line 8, in <module>
    from .basic import Booster, Dataset, register_logger
  File "/Users/elise/Software/miniconda3/envs/ensem/lib/python3.8/site-packages/fairgbm/basic.py", line 95, in <module>
    _LIB = _load_lib()
  File "/Users/elise/Software/miniconda3/envs/ensem/lib/python3.8/site-packages/fairgbm/basic.py", line 86, in _load_lib
    lib = ctypes.cdll.LoadLibrary(lib_path[0])
  File "/Users/elise/Software/miniconda3/envs/ensem/lib/python3.8/ctypes/__init__.py", line 451, in LoadLibrary
    return self._dlltype(name)
  File "/Users/elise/Software/miniconda3/envs/ensem/lib/python3.8/ctypes/__init__.py", line 373, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen(/Users/elise/Software/miniconda3/envs/ensem/lib/python3.8/site-packages/fairgbm/lib_lightgbm.so, 0x0006): tried: '/Users/elise/Software/miniconda3/envs/ensem/lib/python3.8/site-packages/fairgbm/lib_lightgbm.so' (not a mach-o file), '/System/Volumes/Preboot/Cryptexes/OS/Users/elise/Software/miniconda3/envs/ensem/lib/python3.8/site-packages/fairgbm/lib_lightgbm.so' (no such file), '/Users/elise/Software/miniconda3/envs/ensem/lib/python3.8/site-packages/fairgbm/lib_lightgbm.so' (not a mach-o file)
>>>

My operation system is macOS Ventura 13.4, and I have tried

brew install libomp
brew install open-mpi

Everything I found on the Internet is for lightgbm, and nearly nothing is relevent to the similar issue of fairgbm. I also tried deleted the previous conda environment and created new ones, but it didn't work. So I was wondering if anyone may face the same issue or happen to know what I should do next to fix this issue? Thanks in advance

André Cruz · Answer 7 · Thu Sep 21 2023 16:03:33 GMT+0800 (China Standard Time)

Hi @eustomaqua

Unfortunately compatibility with macOS is not maintained. Please try to use a linux machine.
For example, Google Colab is free and works fine with fairgbm.

Here's an example colab notebook that trains a few FairGBM models with hyperparameter-tuning: https://colab.research.google.com/drive/1OElgST6yLpcgsC2i4BE_dJxVny_bLN6F?usp=sharing

Yijun Bian · Answer 8 · Thu Sep 21 2023 19:49:56 GMT+0800 (China Standard Time)

@AndreFCruz Thanks for your reply!

I actually tried using docker (continuumio/anaconda3) as well, but it didn't work

(envs) root@addc1df49268:~# python
Python 3.8.18 (default, Sep 11 2023, 13:19:25)
[GCC 11.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import lightgbm
>>> import fairgbm
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/conda/envs/ensem/lib/python3.8/site-packages/fairgbm/__init__.py", line 8, in <module>
    from .basic import Booster, Dataset, register_logger
  File "/opt/conda/envs/ensem/lib/python3.8/site-packages/fairgbm/basic.py", line 95, in <module>
    _LIB = _load_lib()
  File "/opt/conda/envs/ensem/lib/python3.8/site-packages/fairgbm/basic.py", line 86, in _load_lib
    lib = ctypes.cdll.LoadLibrary(lib_path[0])
  File "/opt/conda/envs/ensem/lib/python3.8/ctypes/__init__.py", line 451, in LoadLibrary
    return self._dlltype(name)
  File "/opt/conda/envs/ensem/lib/python3.8/ctypes/__init__.py", line 373, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /opt/conda/envs/ensem/lib/python3.8/site-packages/fairgbm/lib_lightgbm.so: cannot open shared object file: No such file or directory
>>>
[1]+  Stopped                 python

So I was wondering perhaps using docker was still somehow dependent on the local operation system? But I didn't figure it out yet.

And thank you for sharing the colab example with me, but I'm using fairgbm as a baseline method for my project, so I feel it would be more convenient for me to use it on our server/local environment.

Yijun Bian · Answer 9 · Thu Sep 21 2023 22:30:23 GMT+0800 (China Standard Time)

I built a docker image myself and it works now. Here is my Dockerfile

FROM --platform=linux/amd64 ubuntu:latest

RUN apt-get update && apt-get install -y wget
RUN apt-get update && apt-get install vim -y

RUN wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
RUN mv Miniconda3-latest-Linux-x86_64.sh /root/

Then run this built image and install miniconda3 and other packages including fairgbm, and it works

André Cruz · Answer 10 · Fri Sep 22 2023 17:03:50 GMT+0800 (China Standard Time)

Thanks @eustomaqua !

I'll check it out and add it as an option for non-linux users.

André Cruz · Answer 11 · Fri Sep 22 2023 22:21:17 GMT+0800 (China Standard Time)

This dockerfile should work for non-linux users: https://github.com/AndreFCruz/fairgbm-fork/blob/add-colab-example/examples/FairGBM-python-notebooks/Dockerfile

Image is public and can be ran with docker run -p 8888:8888 ndrcrz/fairgbm-miniconda