Segmentation fault while opening files with cfgrib

[Daniel Lassahn]

I am not able to open grib file with cfgrib . I am running into a segmentation fault.

I have tested both:

xarray.open_dataset('test.grib2', engine='cfgrib')
cfgrib.open_datasets('test.grib2')

test.zip

When I am using pygrib, all works fine. So I assume a problem with cfgrib.

Versions:

In [10]: cfgrib.__version__
Out[10]: '0.9.10.2'

In [11]: xarray.__version__
Out[11]: '2022.9.0'

In [12]: #xarray.__version__

In [13]: import eccodes

In [14]: eccodes.__version__
Out[14]: '2.27.0'

[Daniel Lassahn]

With version: 0.9.8.4 all works fine.

[Iain Russell]

Interesting. For me, cfgrib 0.9.10.2 works fine with the data. My only difference from you is that my test environment has eccodes 2.26.0 rather than 2.27.0, otherwise the versions are the same as your original failing one.

[Daniel Lassahn]

Thanks @iainrussell , I will check out other versions.

Maybe it has something to do with one of the dependecies?

Here is my Dockerfile:

FROM python:3.9.1-slim
MAINTAINER Daniel Lassahn <[e-mail]>

ENV DEBIAN_FRONTEND noninteractive
ENV TERM linux

RUN set -ex \
    && apt update \
    && apt install -y apt-transport-https curl


WORKDIR /tmp
RUN set -ex \
    && buildDeps=' \
        libnetcdf-dev \
        libhdf5-dev \
        build-essential \
        libbz2-dev \
    ' \
    && apt-get update -yqq \
    && apt-get install -yqq --no-install-recommends \
        $buildDeps \
        cdo \
        curl \
        libeccodes0 \
        wget \
        bzip2 \
    && apt-get remove -y $buildDeps \
    && apt-get autoremove -y


RUN wget https://cmake.org/files/v3.20/cmake-3.20.1.tar.gz
RUN tar -xzvf cmake-3.20.1.tar.gz
WORKDIR cmake-3.20.1/
RUN apt-get update && apt-get install --fix-missing -yqq libssl-dev build-essential
RUN ./bootstrap
RUN make -j$(nproc) && make install

RUN apt-get install -y gfortran protobuf-compiler libaec-dev
WORKDIR /
RUN curl https://confluence.ecmwf.int/download/attachments/45757960/eccodes-2.27.0-Source.tar.gz?api=v2 -o /tmp/eccodes-2.27.0-Source.tar.gz
RUN tar -xzf /tmp/eccodes-2.27.0-Source.tar.gz
RUN mkdir build \
    && cd build \
    && cmake /eccodes-2.27.0-Source -DCMAKE_INSTALL_PREFIX=/usr/local/share/eccodes \
    && make install

COPY ./requirements.txt /opt/requirements.txt
RUN pip3 install -r /opt/requirements.txt

[Iain Russell]

Hi @meteoDaniel,
The only odd thing I see there is the installation of eccodes0 from apt-get and then building and installing a version into /usr/local/share/eccodes. I wonder if that self-built installation could be overwriting the definition files installed by eccodes0? That would not explain why it works with a previous version of cfgrib though. But it can be useful to check the eccodes command-line tools, e.g. grib_ls test.grib2 to see if they work.

[Andreas Motl]

Dear Daniel,

does using python:3.9-slim-bullseye make any difference? It will give you a more recent version of Python 3.9, and a newer OS baseline image (Debian 11 Bullseye instead of Debian 10 Buster).

With kind regards,
Andreas.

[Daniel Lassahn]

So I have investigated that removing eccodes0 from the preliminary build deps has no effect. At the beginning of cfgrib it was recommended part of the installation.
So the behaviour really changed with cfgrib version.

With version: cfgrib==0.9.8.5 I receive another issue:

    raise RuntimeError(f"ecCodes library not found using {LIBNAMES}")
E   RuntimeError: ecCodes library not found using ['/usr/local/share/eccodes/lib64/libeccodes.so', '/usr/local/share/eccodes/lib/libeccodes.so', 'eccodes', 'libeccodes.so', 'libeccodes']
________ ERROR collecting tests/modules/parsing/test_parse_grib_data.py ________
/usr/local/lib/python3.9/site-packages/cfgrib/messages.py:42: in <module>
    import pyeccodes.compat as eccodes
E   ModuleNotFoundError: No module named 'pyeccodes'

And from Version 0.9.9.0 I am running into Segmentation fault.

@amotl I will go forward with your suggestion . Thanks ;)

Do I have the right CMake version @iainrussell ?

[Daniel Lassahn]

Using python:3.9-slim-bullseye works fine. Thanks @amotl for his hint.

Seems that I was running into an edge case, so I close this issue because there is a suitable work around.