Giters

motiwari / BanditPAM

BanditPAM C++ implementation and Python package

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Problem with installing banditPAM

retzerjj opened this issue · comments

commented

I've been following some discussions about install issues however I still can't seem to successfully install banditPAM. I'm getting a long error message that seems to indicate that building the wheel for banditPAM failed (full error attached).
I'm working with a MAC intel using Monterrey. I successfully install llvm and armadillo. To answer some questions asked previously:
When I try pip install ., I get:
ERROR: Directory '.' is not installable. Neither 'setup.py' nor 'pyproject.toml' found.

When I execute gcc --version I get:
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-
dir=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Apple clang version 13.0.0 (clang-1300.0.29.30)
Target: x86_64-apple-darwin21.2.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

When I try pip3 -V I get:
pip 21.3.1 from /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pip (python 3.10)

I've tried multiple install commands (e.g. pip, pip3 etc.) but with no luck. Any suggestions would be greatly appreciated,
Thanks!

banditPAMError.txt

commented

Hi @retzerjj , thanks for filing an issue. Could you try pip install banditpam? This should pull version 2.0.0 from PyPI.

The error message you surfaced (ERROR: Directory '.' is not installable. Neither 'setup.py' nor 'pyproject.toml' found.) also suggests that you may be trying to install the repository from the wrong root directory. If you'd like to install the code from source, could you run pip install --no-cache-dir -e . (note the trailing period) from the repo's root directory, e.g. /BanditPAM?

commented

I also just took a closer look at your full error output and it looks like your LLVM installation is having trouble finding the C++ standard library headers. Did you try to install LLVM from source? If so, could you install it instead via brew install llvm?

More helpful details are here: https://github.com/ThrunGroup/BanditPAM/blob/main/docs/install_mac.md

commented

Thanks very much for your response. I did install llvm using brew. When executing the command again (brew install llvm) I get the following message:

Warning: llvm 13.0.0_2 is already installed and up-to-date.
To reinstall 13.0.0_2, run:
brew reinstall llvm

I next downloaded and extracted banditPAM and ran

pip install --no-cache-dir -e .

in the downloaded directory. It does indeed look like there is a problem finding the stdio.h header file (new error message text attached) but since the llvm installation looks ok, I'm not sure what to do next.

Thanks again very much

Note: not sure if this plays a role however armadillo appears to have been installed correctly too:

brew install armadillo 

   Warning: armadillo 10.7.5 is already installed and up-to-date.

To reinstall 10.7.5, run:
brew reinstall armadillo

banditPAMError2.txt

commented

Thanks @retzerjj ! Just to confirm, did you also install the xcode command line tools via the Mac installation guide? This post suggests that might be the issue

commented

Hi @motiwari, yes I did. When I try

xcode-select --install

I get

xcode-select: error: command line tools are already installed, use "Software Update" to install updates

Thanks again.

commented

Interesting -- could you update the Xcode tools by opening the "App Store" app and Choosing "Updates" on the left-hand sidebar?

Also, does your /usr/local/ directory exist?

commented

Hello @motiwari

I checked and I do have a /usr/local directory. I also checked and there was no update for xcode in the app store app.

I tried to update xcode command line tools (which were installed). I deleted them and re-installed from the command
line. I also noted that when I opened the xcode app it asked to install additional software and I did so.

I then re-booted the computer and after trying to install banditPAM (this time version 3) and it did in fact work!!

Many thanks for all your help!

However, I next went to install banditPAM on my macbook (apple silicon). I made sure I had armadillo/llvm installed and
xcode tools was present and updated. I now get a slightly different error saying, I believe, that it cant find
armadillo. I checked and it appears llvm is installed under homebrew, unlike on my iMac, but my path appears to be
correctly indicating where to look:

echo $PATH
/opt/homebrew/opt/llvm/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/opt/X11/bin:/Library/Apple/usr/bin

I've attached the error message to this note. Again, any suggestions would be greatly appreciated, your help is
invaluable!!

P.S. Two additional notes (not sure if they are material)
1. Apple released an update for command line tools which was installed on both machines
2. The iMac (where banditPAM installed successfully) is Intel based using a bash shell.
The MacBook Air (where the error is now being produced) is Apple M1 Silicon and uses a zsh shell.

Thanks again very much!
banditPAM_error_m1.txt

commented

Hey @retzerjj , that's great to hear! Thanks for reporting back with the fix -- I've added this to the install guide to help future generations.

Regarding the new error on the Macbook Air, could you confirm you've run brew install armadillo and provide the output of brew --prefix armadillo?

Also, is there a file called armadillo in your /usr/local/include? If not, we may need to create a symlink there to your brew-installed armadillo (perhaps the brew link step did not complete successfully and raised an error when installing armadillo?). For example, here is the output on my machine:

>>>brew --prefix armadillo
/usr/local/opt/armadillo
>>>ls -alh /usr/local/include/armadillo
lrwxr-xr-x 1 motiwari admin 44 Nov 11 09:04 /usr/local/include/armadillo -> /usr/local/Cellar/armadillo/10.7.3/include/armadillo

By the way, the PATH variable is only used to search for executables and so won't help the linking problem. Instead, we may need to edit your LD_LIBRARY_PATH, which tells the linker where to look for libraries it needs, like armadillo.

commented

Oh, it also looks like M1 Macs have changed some important directories. Thanks for surfacing this bug, I'll work on a fix!

In the meantime, you can likely get it to work by creating a symlink in a (new) /usr/local/include directory for armadillo, pointing to where brew has installed it

commented

@retzerjj could you try the following command and let me know if it works? You may need a sudo:

sudo python3 -m pip install -e git+https://github.com/ThrunGroup/BanditPAM.git@m1_macs#egg=banditpam

commented

Hello @motiwari ,
I tried your suggestions and my results are listed below

  1. Here is the path I get when I execute "brew --prefix armadillo"
    /opt/homebrew/opt/armadillo

  2. I checked and there is no file called armadillo in "/usr/local/include"

  3. I tried adding to my path in ~/.zshrc (based on recommendations from the link regarding M1 Mac important directory changes):
    export HOMEBREW_PREFIX="/opt/homebrew"
    export HOMEBREW_CELLAR="/opt/homebrew/Cellar"
    export HOMEBREW_NO_ANALYTICS=1

  4. I also tried creating a symlink with the command "sudo ln -s /opt/homebrew/opt /usr/local/include"
    (which may or may not be correct). I was unable to compile banditPAM after this as well so I unlinked it.

  5. I executed the command
    "sudo python3 -m pip install -e git+https://github.com/ThrunGroup/BanditPAM.git@m1_macs#egg=banditpam"
    The output in attached file "newInstallAttempt.txt"

Thanks again for all your help, Happy New Year!

newInstallAttempt.txt

commented

Happy New Year, @retzerjj !

Could you try running the command below again? I just updated something.

sudo python3 -m pip install -e git+https://github.com/ThrunGroup/BanditPAM.git@m1_macs#egg=banditpam

If that still doesn't work, could you try running the following command

mkdir -p /usr/local/include && ln -s /opt/homebrew/opt/armadillo/include/armadillo /usr/local/include/armadillo

and then running pip install banditpam again?

If neither of those work, could you provide the output of the following command?

tree /opt/homebrew/opt/armadillo

A few tips that might help:

  • The problem is that on M1 Macs, the installation location of packages like armadillo has changed
  • If it is easier for you, you can provide your terminal output in pastes from PasteBin
  • You can use single backtick (the key to the left of the 1) for code formatting and triple backticks for
code block
commented

Hello @motiwari
I am still unable to install banditPAM. It seems like the error is not related to not finding armadillo in /usr/local/include but rather maybe that an error is generated once it is found:

#include "armadillo_bits/config.hpp"
^~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
error: command '/opt/homebrew/opt/llvm/bin/clang' failed with exit code 1

I also ran the tree command, output is in the attached txt file.
Thanks!

treeOutput.txt

commented

@retzerjj We are making progress! Can you try one more time:

sudo python3 -m pip install -e git+https://github.com/ThrunGroup/BanditPAM.git@m1_macs#egg=banditpam

commented

@motiwari
it looks like its not getting past the setup.py step (error attached
newError.txt
)

commented

Oops sorry, can you try this command instead?

sudo python3 -m pip install --no-cache-dir -e git+https://github.com/ThrunGroup/BanditPAM.git@m1_macs#egg=banditpam

commented

Looks like a similar error
newError2.txt
.

commented

Sorry for the churn here, @retzerjj . Let me get my hands on an M1 Mac so I can debug locally and save you the time. I'll get back to you shortly

commented

Thanks very much @motiwari, I very much appreciate it.

commented

Hey @retzerjj , just letting you know I haven't forgotten about this. It turns out building for M1 Macs is much harder than I initially thought, but I'm making progress and should have an update soon

commented
commented

Hi @retzerjj ; I think I was able to build the Python wheel for M1 Macs; could you install banditpam via pip install -i https://test.pypi.org/simple/ banditpam==3.0.3a1 and let me know if it works?

The code in Example 1 should run.

A caveat: you'll have to use CPython 3.8+. I had a friend with an M1 Mac test it and it appears to work!

In case you get an error like RuntimeError: module compiled against API version 0xf but this version of numpy is 0xe, you may need to run pip install --upgrade numpy

commented
commented

Thanks @retzerjj ! It's strange; pip install -i https://test.pypi.org/simple/ banditpam==3.0.3a1 should not have succeeded unless you're using CPython. In the pip output, did it somehow fall back to a previous version? What's the output if you run python -c "import platform; print(platform.python_implementation())"?

CPython should be the default Python implementation when using Anaconda, which I highly recommend!

If you're running CPython, the version is >=3.8, and you're still getting this error, could you provide the output of the following commands?

a) the output when you open a python shell:

>>> python
Python 3.8.12 | packaged by conda-forge | (default, Oct 12 2021, 21:50:38)
[Clang 11.1.0 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.

b) the output of pip -V

commented
commented
commented

Hmmm it seems Sudhir deleted his comment, but we owe a thank you to him!

@retzerjj you can install Python from Anaconda here; it should provide instructions on how to activate the appropriate Anaconda environment after you've installed it. By the way, did you copy-paste the wrong information in this comment?

Probably it works on Anaconda because I did at least something right when creating the M1 Mac wheels; but I've also heard from others that the wheels don't work with Miniconda or other Python installations. I'll leave this issue open until the package is working everywhere

commented

Yes, Banditpam was successfully installed, but I was unable to use it because it displayed an error while importing it.

"ImportError: dlopen(/Users/sudhirwork/anaconda3/lib/python3.9/site-packages/banditpam.cpython-39-darwin.so, 0x0002): symbol not found in flat namespace '__ZTWN4arma19mt19937_64_instanceE'"

commented
commented

Sorry for the late response @retzerjj . I'm not sure how to attach pictures by email-replying to Github email notifications, but if you sign into Github and view this link, you can attach an image in the comment box.

At the moment, I'm not sure what to do for this because I can't reproduce your or Sudhir's error locally without an M1 Mac. Two of my friends with M1 Mac where able to use the M1 wheel successfully, but another one surfaced an error like this:

ImportError: dlopen(/Users/frieda/miniconda3/envs/behavior/lib/python3.8/site-packages/banditpam.cpython-38-darwin.so, 0x0002): symbol not found in flat namespace '_omp_get_max_threads'

This looks like this is different from Sudhir's error message, which seems to be complaining about an armadillo symbol.

I ordered an M1 Mac on 12/29/21, but it still won't be delivered until 3/10/22 (presumably due to existing supply chain issues). If you provide your error message then I can investigate it when I receive the M1 Mac

commented
commented

Hi @retzerjj , my apologies for the late response. I've still been unable to acquire an M1 Mac to test this locally, so to unblock you I've deleted the offending functions and shipped a new version to PyPI.

When the new builds for v3.0.4 hit PyPI in about an hour, you should be able to run pip install --upgrade banditpam and upgrade to a version runnable on M1 Macs.

If the issue persists, please let me know and I'll take a look.

commented
commented

@retzerjj sorry for the very long delay here. I've revisited this issue with the release of BanditPAM v4.0.0. Does this issue still persist?

If so, could you try installing BanditPAM inside an Anaconda environment and let me know if you still have the same error?

commented
commented

Hey Joe, the output of your first command (pip install banditpam) shows that you currently have BanditPAM v3.0.4 installed. If you run pip install --upgrade banditpam and try to run a python script that runs import banditpam, what output do you get?

commented
commented

Thanks @retzerjj . Unfortunately I spent several days trying to debug this issue and wasn't able to fix it. I'm not sure how this could possibly be happening, as the necessary libraries like OpenMP are statically linked within the Python wheel.

It seems that other users are experiencing the same issue (#230 ); if they are able to find a solution, I'll update this thread.

commented

Note to self: ContinuumIO/anaconda-issues#12933 looks relevant

commented

I got the same error ImportError: /root/venv/lib/python3.9/site-packages/banditpam.cpython-39-x86_64-linux-gnu.so: undefined symbol: _ZNSt19basic_ostringstreamIcSt11char_traitsIcESaIcEEC1Ev on Deepnote .
Screenshot 2023-05-01 at 3 00 51 PM

The banditpam package is installed via requirements.txt

commented

Thanks @thucnc, I'll let you know if I find a solution. @Adarsh321123 could you try installing banditpam on a Deepnote workbook and see if you're able to reproduce the result?

commented

Sorry for the late response, @motiwari. I am able to reproduce the result. Specifically, after installing the banditpam package via requirements.txt as @thucnc mentioned, I ran a cell with the Python code from banditpam import KMedoids and got this exception: ImportError: /root/venv/lib/python3.9/site-packages/banditpam.cpython-39-x86_64-linux-gnu.so: undefined symbol: _ZNSt19basic_ostringstreamIcSt11char_traitsIcESaIcEEC1Ev.

commented

Hi @retzerjj @SumanSudhir @thucnc I believe we have finally diagnosed the source of the problem.

TL;DR: Could you please install Anaconda from https://github.com/conda-forge/miniforge#miniforge3 specifically for arm64 (Apple Silicon)? It's critical to install from the link in the screenshot below. After you have changed your Python installation, pip install banditpam should work and you should be able to run the examples in the README.

image

More context: We realized that the default installer for Anaconda, even on M1 Macs, runs in x86_64 mode (likely using Rosetta). This causes a problem at runtime because the libraries installed for arm64 machines via brew install libomp armadillo are the arm64 versions on M1 Macs, if you're using the appropriate version of brew for M1 Macs. Because the default installer Anaconda for M1 Macs runs in x86_64 mode, it tries to download the x86_64 version of python packages via pip (see this comment which explicitly shows the x86_64 version of the package being installed). At runtime, it's unable to dynamically load the arm64 versions of the necessary libraries like LibOMP and Armadillo. The situation should be fixed by downloading and using the arm64 version of Anaconda, which should also download the arm64 versions of the banditpam python wheels.

commented

Hi, I'm running into a similar issue. I'm not using anaconda, I'm just using pyenv/python.

❯ python
Python 3.10.7 (main, Nov  2 2022, 15:27:56) [Clang 14.0.0 (clang-1400.0.29.202)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import platform
>>> platform.machine()
'arm64'
>>> import banditpam
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: dlopen(..../.venv/lib/python3.10/site-packages/banditpam.cpython-310-darwin.so, 0x0002): symbol not found in flat namespace '_omp_get_max_threads'
>>>
commented

Hi @li-dennis, could you provide the output of the following two commands?

brew --prefix armadillo and python -m pip install --ignore-installed --no-cache-dir banditpam?

commented

Sorry, just seeing this :) sure

❯ brew --prefix armadillo
/opt/homebrew/opt/armadillo
❯ python -m pip install --ignore-installed --no-cache-dir banditpam
Collecting banditpam
  Downloading banditpam-4.0.2-cp310-cp310-macosx_11_0_arm64.whl (241 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 241.8/241.8 kB 2.3 MB/s eta 0:00:00
Installing collected packages: banditpam
Successfully installed banditpam-3.0.4
commented

Something is weird about that output -- it looks like you're downloading v4.0.2 but then the output says you installed banditpam-3.0.4.

In any case, I think we know the root cause of the problem and are trying to fix.

commented

Ah, yeah, my bad -- was messing around with pipenv and v3.0.4 as well to see if the fault lied there so there was some version mismatching

❯ python -m pip install --ignore-installed --no-cache-dir banditpam

Collecting banditpam
  Downloading banditpam-4.0.2-cp310-cp310-macosx_11_0_arm64.whl (241 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 241.8/241.8 kB 2.1 MB/s eta 0:00:00
Installing collected packages: banditpam
Successfully installed banditpam-4.0.2
❯ python
Python 3.10.7 (main, Nov  2 2022, 15:27:56) [Clang 14.0.0 (clang-1400.0.29.202)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import banditpam
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: dlopen(.../.venv/lib/python3.10/site-packages/banditpam.cpython-310-darwin.so, 0x0002): symbol not found in flat namespace '_omp_get_max_threads'