Problem on installation 1.3.1 with sdist

Question

Problem on installation 1.3.1 with sdist

YuujinHwang opened this issue 2 years ago · comments

Environment
macOS Monterey
Version 12.1
Chip Apple M1 Max
Python 3.10.1

With the goes library as below

brew install geos
export $GEOS_DIR=/opt/homebrew/Cellar/geos/3.10.2/
/opt/homebrew/Cellar/geos/3.10.2/

I tried to install basemap following the comments on #531

poetry add basemap
# same symptom with pip

but installation failed with error

EnvCommandError

  Command ['/Users/yuujin/Workspace/.venv/bin/pip', 'install', '--no-deps', 'file:///Users/yuujin/Library/Caches/pypoetry/artifacts/b1/eb/1a/6b80b1e92c9b7b023c7d118d790f51ffc701b0c0e5f1e87b6959ab1b89/basemap-1.3.1.zip'] errored with the following return code 1, and output: 
  Processing /Users/yuujin/Library/Caches/pypoetry/artifacts/b1/eb/1a/6b80b1e92c9b7b023c7d118d790f51ffc701b0c0e5f1e87b6959ab1b89/basemap-1.3.1.zip
    Preparing metadata (setup.py): started
    Preparing metadata (setup.py): finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /Users/yuujin/Workspace/.venv/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/5r/0p5dh9vx78qbptqmcm3zmgdm0000gn/T/pip-req-build-diy_eywu/setup.py'"'"'; __file__='"'"'/private/var/folders/5r/0p5dh9vx78qbptqmcm3zmgdm0000gn/T/pip-req-build-diy_eywu/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/5r/0p5dh9vx78qbptqmcm3zmgdm0000gn/T/pip-pip-egg-info-548b08zr
         cwd: /private/var/folders/5r/0p5dh9vx78qbptqmcm3zmgdm0000gn/T/pip-req-build-diy_eywu/
    Complete output (9 lines):
    /private/var/folders/5r/0p5dh9vx78qbptqmcm3zmgdm0000gn/T/pip-req-build-diy_eywu/setup.py:52: RuntimeWarning: Cannot find GEOS library and/or headers in standard locations ('/opt/homebrew/Cellar/geos/3.10.2/'). Please install the corresponding packages using your software management system or set the environment variable GEOS_DIR to point to the location where GEOS is installed (for example, if 'geos_c.h' is in '/usr/local/include' and 'libgeos_c' is in '/usr/local/lib', then you need to set GEOS_DIR to '/usr/local'
      warnings.warn(" ".join([
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/5r/0p5dh9vx78qbptqmcm3zmgdm0000gn/T/pip-req-build-diy_eywu/setup.py", line 149, in <module>
        dev_requires = get_content("requirements-dev.txt", splitlines=True)
      File "/private/var/folders/5r/0p5dh9vx78qbptqmcm3zmgdm0000gn/T/pip-req-build-diy_eywu/setup.py", line 23, in get_content
        with io.open(path, encoding="utf-8") as fd:
    FileNotFoundError: [Errno 2] No such file or directory: '/private/var/folders/5r/0p5dh9vx78qbptqmcm3zmgdm0000gn/T/pip-req-build-diy_eywu/requirements-dev.txt'
    ----------------------------------------
  WARNING: Discarding file:///Users/yuujin/Library/Caches/pypoetry/artifacts/b1/eb/1a/6b80b1e92c9b7b023c7d118d790f51ffc701b0c0e5f1e87b6959ab1b89/basemap-1.3.1.zip. Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Even there exist header and built binary 'geos_c.h' and 'libgeos_c' in $GEOS_DIR

Víctor Molina García · Answer 1 · Thu Feb 10 2022 15:07:05 GMT+0800 (China Standard Time)

Hi @YuujinHwang!

You are right. Because we do not have pre-built wheels for MacOS, your computer tries with the source distribution (the zip file) but I made a mistake and some files are missing in the zip file uploaded to PyPI (see #532). This is fixed but I haven't released basemap 1.3.2 yet.

Can you try to install basemap using the following sdist (this is the current development status):
basemap-1.3.1+dev.zip

If it works for you, then I will release basemap 1.3.2 today and in the future you can install it with pip as you were trying to do.

Also I see that you are trying to link basemap with GEOS 3.10.2. There are known problems with some drawing using newer GEOS (see #522), so I would recommend you to link basemap with GEOS < 3.9, which is known to work well.

Víctor Molina García · Answer 2 · Thu Feb 10 2022 23:57:42 GMT+0800 (China Standard Time)

I have just released basemap 1.3.2 that includes the fix for the issue with the source distribution that you were facing. Could you try to repeat the installation with the same command as before? Now it should work:

python -m pip install basemap

Víctor Molina García · Answer 3 · Tue Feb 15 2022 00:13:20 GMT+0800 (China Standard Time)

@YuujinHwang May I ask you if your problem got solved by installing basemap 1.3.2 as indicated in my previous message?

Sourish Basu · Answer 4 · Tue Feb 22 2022 06:03:20 GMT+0800 (China Standard Time)

@molinav Is issue #522 also fixed by this new release?

Víctor Molina García · Answer 5 · Tue Feb 22 2022 15:31:54 GMT+0800 (China Standard Time)

@AgilentGCMS Your issue is still not solved in basemap 1.3.2, but I keep it in my backlog to study it when I get some time. The workaround for the moment is still to pin GEOS < 3.9.

Víctor Molina García · Answer 6 · Tue Mar 01 2022 19:34:15 GMT+0800 (China Standard Time)

I am closing this issue due to lack of feedback. Since this was a problem with the broken sdist in PyPI and this is solved in basemap 1.3.2, this issue should be solved with this release. If that is not the case, feel free to reopen the issue.

Guorun · Answer 7 · Thu Apr 07 2022 11:02:01 GMT+0800 (China Standard Time)

Hi! I am working on Macbook pro with basemap, but I met some issues. When I try to pip install basemap, there is something wrong with this. But I have already brew install geos, what's wrong? Thanks!

@YuujinHwang @molinav

Víctor Molina García · Answer 8 · Sat Apr 16 2022 18:16:05 GMT+0800 (China Standard Time)

@YuujinHwang The fix that I made to solve the broken source distribution package was not enough to make basemap installable on MacOS. I needed to make another patch as shown in #539 to ensure that the GEOS shared library (dylib) can be found. May I ask you to check if the fix also works for your case? You would need to do the following:

export GEOS_DIR="/opt/homebrew/Cellar/geos/3.10.2"
git clone https://github.com/matplotlib/basemap.git
cd basemap/packages/basemap
git checkout bugfix-539
python -m pip install .

Oscar · Answer 9 · Sun Apr 17 2022 18:14:50 GMT+0800 (China Standard Time)

@molinav Hey! This got me past the install step (M1 Pro), with an extra
python3 -c "import utils; utils.GeosLibrary('3.5.1').build(installdir='${GEOS_DIR}')"
step in the middle. All working now 🚀

Python 3.10.2, pip 22.0.4

Víctor Molina García · Answer 10 · Sun Apr 17 2022 20:39:14 GMT+0800 (China Standard Time)

@odrusso Thanks for the feedback! You have to use your additional command in case that you also need to build GEOS from source. I didn't include it in my previous comment because the original issue was trying to link the GEOS library available from Homebrew. Good to see that with GEOS from source it also works flawlessly on MacOS.

I will tag a new basemap patch version (1.3.3) very soon. Then it will be possible to install basemap on MacOS using the source distributions from PyPI; basemap 1.3.3 should therefore work on:

Generic Windows x86 and x64 (prebuilt wheels or from source).
Generic GNU/Linux x86 and x64 (prebuilt wheels or from source).
MacOS x64 and ARM (from source; needs C compiler available, setting GEOS_DIR environment variable, and CMake if also building GEOS from source).
Cygwin/MinGW64 Windows (from source; needs C compiler available, and setting GEOS_DIR environment variable, and GEOS from their package managers).

Víctor Molina García · Answer 11 · Sun Apr 17 2022 20:51:55 GMT+0800 (China Standard Time)

@odrusso Thanks for the feedback! You have to use your additional command in case that you also need to build GEOS from source. I didn't include it in my previous comment because the original issue was trying to link the GEOS library available from Homebrew. Good to see that with GEOS from source it also works flawlessly on MacOS.

I will tag a new basemap patch version (1.3.3) very soon. Then it will be possible to install basemap on MacOS using the source distributions from PyPI; basemap 1.3.3 should therefore work on:

Generic Windows x86 and x64 (prebuilt wheels or from source).
Generic GNU/Linux x86 and x64 (prebuilt wheels or from source).
MacOS x64 and ARM (from source; needs C compiler available, setting GEOS_DIR environment variable, and CMake if also building GEOS from source).
Cygwin/MinGW64 Windows (from source; needs C compiler available, and setting GEOS_DIR environment variable, and GEOS from their package managers).

極貓 (J. Moh) · Answer 12 · Tue Jun 28 2022 12:14:53 GMT+0800 (China Standard Time)

I have just released basemap 1.3.2 that includes the fix for the issue with the source distribution that you were facing. Could you try to repeat the installation with the same command as before? Now it should work:
python -m pip install basemap

hi @molinav , I'm facing the same problem as the issue creator does. My OS is macOS 12.4. and the version 1.3.2 and 1.3.3 can not install as well, the reported final error is below,

              File "/private/var/folders/95/d5hdsfz15nv_ljrcg3l51pt00000gn/T/pip-install-9hm5xce1/numpy_4b6eeeaaaff94e648df3ce8efda8c580/numpy/distutils/fcompiler/gnu.py", line 144, in get_flags_linker_so
                os.environ['MACOSX_DEPLOYMENT_TARGET'] = target
              File "/usr/local/Cellar/python@3.9/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/os.py", line 684, in __setitem__
                value = self.encodevalue(value)
              File "/usr/local/Cellar/python@3.9/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/os.py", line 756, in encode
                raise TypeError("str expected, not %s" % type(value).__name__)
            TypeError: str expected, not int

Víctor Molina García · Answer 13 · Tue Jun 28 2022 15:23:40 GMT+0800 (China Standard Time)

Hi @junchih! It looks like you might having the following issue with numpy: numpy/numpy#18093

Are you trying to build numpy from source? It could also be that your system already had numpy installed but with a version prior to the patch from the link above.

極貓 (J. Moh) · Answer 14 · Tue Jun 28 2022 22:09:08 GMT+0800 (China Standard Time)

Thx @molinav, I see the reason.

Theoretically anybody with macOS builtin python 3.8, or brew install python@3 (python 3.9) will be punched by this error. Because of https://github.com/matplotlib/basemap/blob/v1.3.2/packages/basemap/pyproject.toml#L5-L6.

Only macOS with python 3.10 could work. But Homebrew/homebrew-core#91421. Let's wait!

Víctor Molina García · Answer 15 · Wed Jun 29 2022 15:53:07 GMT+0800 (China Standard Time)

The basemap series 1.3.x is aiming to put basemap in shape for proper distribution under as many operating systems and Python versions as possible.

Unfortunately I do not have expertise with MacOS and this might have introduced some regressions. The lines that you mention are very useful for an eventual fix for Python 3.8+ on MacOS. The reason for fixing numpy to old versions (e.g. 1.16.6) in the build dependencies was to ensure that the generated wheels for PyPI would work for anybody independently of the runtime numpy version (old ones and new ones). If the wheels were built linking to a too new numpy version (especially with numpy >= 1.20), this would cause errors (or at least warnings) to users with numpy < 1.20 at runtime.

But based on what we have seen here, this is a no-go for MacOS users due to numpy/numpy#18093, which was never fixed for numpy version 1.16.6. So the approach of using "the-oldest-numpy-possible" seems ok for Windows and GNU/Linux, but for MacOS we need to relax it (>= instead of ==).

This issue is also a clear example of the importance of implementing a GitHub workflow for MacOS. Implementations for Windows and GNU/Linux are already available and building the official wheels for PyPI. The MacOS workflow is not implemented as of today and errors like the one that you addressed are only found when an end user shares it in the issue tracker.

Víctor Molina García · Answer 16 · Wed Jun 29 2022 16:00:39 GMT+0800 (China Standard Time)

@junchih Does it work for you if installing with the --no-use-pep517 flag, e.g. python -m pip install --no-use-pep517 basemap?

極貓 (J. Moh) · Answer 17 · Wed Jun 29 2022 19:38:11 GMT+0800 (China Standard Time)

@molinav, I got this error, and same error for python 3.9

$ python -m pip install --no-use-pep517 basemap
Collecting basemap
  Using cached basemap-1.3.3.zip (155 kB)
ERROR: Disabling PEP 517 processing is invalid: project specifies a build backend of setuptools.build_meta in pyproject.toml
$ python --version
Python 3.8.9

Víctor Molina García · Answer 18 · Sat Aug 06 2022 23:53:40 GMT+0800 (China Standard Time)

@junchih Your issue was also mentioned in #547 and you were right with the source of the problem. The patch is already implemented in the develop branch, so you can try out by doing:

python -m pip install https://github.com/matplotlib/basemap/archive/refs/heads/develop.zip#subdirectory=packages/basemap

and check that it works with:

python -c "from mpl_toolkits import basemap; print(basemap)"

The patch will arrive in PyPI with the hotfix version 1.3.4 of basemap, which I hope to prepare soon.