Cannot find the ecCodes library when pip installing eccodes 1.3.X

Question

Cannot find the ecCodes library when pip installing eccodes 1.3.X

pmav99 opened this issue 3 years ago · comments

Reproducer

conda create -y  --name test_eccodes python=3.8 pip eccodes
conda activate test_eccodes
python -mvenv venv
source venv/bin/activate
pip install eccodes==1.3.0 # or 1.3.1 or 1.3.2
python -m eccodes selfcheck

results in

Traceback (most recent call last):
  File "/home/panos/.conda/envs/test_eccodes/lib/python3.8/runpy.py", line 185, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/home/panos/.conda/envs/test_eccodes/lib/python3.8/runpy.py", line 144, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "/home/panos/.conda/envs/test_eccodes/lib/python3.8/runpy.py", line 111, in _get_module_details
    __import__(pkg_name)
  File "/home/panos/venv_eccodes/lib/python3.8/site-packages/eccodes/__init__.py", line 15, in <module>
    from .eccodes import *
  File "/home/panos/venv_eccodes/lib/python3.8/site-packages/eccodes/eccodes.py", line 12, in <module>
    from gribapi import __version__
  File "/home/panos/venv_eccodes/lib/python3.8/site-packages/gribapi/__init__.py", line 13, in <module>
    from .gribapi import *  # noqa
  File "/home/panos/venv_eccodes/lib/python3.8/site-packages/gribapi/gribapi.py", line 32, in <module>
    from .bindings import ENC, ffi, lib
  File "/home/panos/venv_eccodes/lib/python3.8/site-packages/gribapi/bindings.py", line 33, in <module>
    raise RuntimeError("Cannot find the ecCodes library")
RuntimeError: Cannot find the ecCodes library

Downgrading to 1.2.0 works just fine

$ pip install eccodes=1.2.0
$ python -m eccodes selfcheck

Found: ecCodes v2.21.0.
Your system is ready.

The eccodes package from conda is eccodes conda-forge/linux-64::eccodes-2.21.0-he2bb022_1

Cleaning up:

deactivate
conda deactivate
rm -rf venv
conda env remove --name test_eccodes

Xunius · Answer 1 · Wed Apr 28 2021 16:16:44 GMT+0800 (China Standard Time)

Same here. pip install eccodes==1.2.0 worked for me.

Christopher Cassidy · Answer 2 · Fri May 21 2021 01:40:54 GMT+0800 (China Standard Time)

Can confirm what the above posters found as well, works with 1.2.0 but fails with anything 1.3.x
Using Python3.8 w/ ecCodes 2.22.0

Christopher Cassidy · Answer 3 · Mon Aug 23 2021 08:16:15 GMT+0800 (China Standard Time)

Hi @pmav99 @Xunius I just did a fresh install of ecCodes 2.22.1 and eccodes-python 1.3.3 on Ubuntu 20.04 and the import worked fine. I saw v1.3.3 was released since we noticed this error maybe it was fixed in that version?

Christopher Cassidy · Answer 4 · Tue Aug 24 2021 04:40:14 GMT+0800 (China Standard Time)

Tried some more tests and there appears to be some sort of conflict on RHEL-based systems: Amazon Linux 2 for example puts the libraries in a lib64/ directory (which fails) while Ubuntu does lib/ (which works).

As a workaround I renamed lib64 to lib and added the lib path to LD_LIBRARY_PATH and it worked under AL2. Not idea but maybe that helps track down the root cause?

Philippe Roy · Answer 5 · Tue Aug 31 2021 03:47:55 GMT+0800 (China Standard Time)

Tried some more tests and there appears to be some sort of conflict on RHEL-based systems: Amazon Linux 2 for example puts the libraries in a lib64/ directory (which fails) while Ubuntu does lib/ (which works).

As a workaround I renamed lib64 to lib and added the lib path to LD_LIBRARY_PATH and it worked under AL2. Not idea but maybe that helps track down the root cause?

Thanks! Updating the LD_LIBRARY_PATH is the only thing that worked for me (cluster on Cent OS)

nfick · Answer 6 · Wed Sep 22 2021 23:25:25 GMT+0800 (China Standard Time)

Changing versions didn't work for me. All I had to do was:
pip install ecmwflibs

From bindings.py in:

File "/home/panos/venv_eccodes/lib/python3.8/site-packages/gribapi/bindings.py", line 33, in <module>
    raise RuntimeError("Cannot find the ecCodes library")

There was:

try:
    import ecmwflibs as findlibs
except ImportError:
    import findlibs

The findlibs package was the problem.

Pete Bachant · Answer 7 · Tue Nov 30 2021 05:23:35 GMT+0800 (China Standard Time)

@nfick's fix works for me on Linux, but on Windows it causes some strange DLL load warnings:

C:\Users\peteb\miniconda3\envs\wedev\lib\site-packages\ecmwflibs\__init__.py:75:
 UserWarning: DLL load failed while importing _ecmwflibs: The specified procedur
e could not be found.
  warnings.warn(str(e))
C:\Users\peteb\miniconda3\envs\wedev\lib\site-packages\xarray\backends\cfgrib_.p
y:27: UserWarning: Failed to load cfgrib - most likely there is a problem access
ing the ecCodes library. Try `import cfgrib` to get the full error message
  warnings.warn(

But import eccodes works without error.

CrayonAki · Answer 8 · Sun Dec 19 2021 21:40:23 GMT+0800 (China Standard Time)

Ubuntu 20.04, python 3.8
Following @nfick, I installed ecmwflibs with pip install ecmwflibs.
I also tried to change version, but eccodes==1.2.0 seemed not working. So I tried version eccodes==1.3.1.
I don't know which step is actually working, but finally eccodes can be imported.

Vipin Karthikeyan · Answer 9 · Thu Dec 23 2021 14:33:25 GMT+0800 (China Standard Time)

Before installing cfgrib,
$ pip install ecmwflibs
$ pip install eccodes==1.3.1
and then
$ pip install cfgrib

my ecmwflibs version is 0.4.3

Thanks to @CrayonAki

Peter Dudfield · Answer 10 · Fri Jan 07 2022 19:19:08 GMT+0800 (China Standard Time)

Im getting a similar error when runner a docker file

FROM python:3.9-slim

RUN apt-get -qq update \
    && apt-get -qq install -y --no-install-recommends \
        libproj-dev \
        libgeos-dev \
    && apt-get autoclean && apt-get autoremove \
    > /dev/null

# copy files
COPY metofficeamd/ app/metofficeamd/
COPY setup.py app/setup.py
COPY README.md app/README.md
COPY requirements.txt app/requirements.txt

# change to app folder
WORKDIR /app

# install library
RUN pip uninstall cfgrib
RUN pip install ecmwflibs
RUN pip install eccodes==1.3.1
RUN pip install cfgrib
RUN pip install -e .

# Example commnad that can be used, need to set API_KEY, API_SECRET and SAVE_DIR
CMD ["python", "metofficeamd/app.py"]

Peter Dudfield · Answer 11 · Fri Jan 07 2022 20:12:19 GMT+0800 (China Standard Time)

Im getting a similar error when runner a docker file

FROM python:3.9-slim

RUN apt-get -qq update \
    && apt-get -qq install -y --no-install-recommends \
        libproj-dev \
        libgeos-dev \
    && apt-get autoclean && apt-get autoremove \
    > /dev/null

# copy files
COPY metofficeamd/ app/metofficeamd/
COPY setup.py app/setup.py
COPY README.md app/README.md
COPY requirements.txt app/requirements.txt

# change to app folder
WORKDIR /app

# install library
RUN pip uninstall cfgrib
RUN pip install ecmwflibs
RUN pip install eccodes==1.3.1
RUN pip install cfgrib
RUN pip install -e .

# Example commnad that can be used, need to set API_KEY, API_SECRET and SAVE_DIR
CMD ["python", "metofficeamd/app.py"]

by adding 'libeccodes-tools' it now seems to work

Eduardo Passeto · Answer 12 · Thu Aug 18 2022 02:10:18 GMT+0800 (China Standard Time)

What solved for me was REALLY to install ECMWFlibs, just because eccodes depends on this earlier library to work. The version of ECcodes does not matter!

So, at Google Colab, you can do:
!$ pip install ecmwflibs
!$ pip install eccodes
!$ pip install cfgrib

Found: eccodes v2.26.0
Your system is ready

David Hoese · Answer 13 · Tue May 09 2023 23:04:03 GMT+0800 (China Standard Time)

I'm working with someone who is trying to add cfgrib/eccodes to our build environment and is running into weird behavior. Sometimes they get the same error(s) mentioned here and other times the errors are slightly different and sometimes it works. Whether the environment works or errors out seems to be related to the order the packages are installed.

The order the packages are installed shouldn't matter, right? There should be no reason you can't do pip install ecmwflibs eccodes cfgrib since all the usage happens at runtime not install time...right?

Note: Some of the issues we're running into may be specific to us trying to use pyinstaller to bundle our environment into a distributed application.