Is it possible to open a GRIB2 message from a `bytes` instance?
milancurcic opened this issue · comments
Hi, I have a GRIB2 message as a byte array in memory (bytes
type). Is it possible to open it with cfgrib without writing it to disk first?
I tried cfgrib.open_dataset
and cfgrib.open_file
. I also tried passing io.BytesIO
instance of my byte array with no success.
This is cfgrib-0.9.10.3 with eccodes-2.16.0 on Ubuntu 20.04 LTS.
Thank you!
I'd also be very interested in this - I have a custom class implementing io.IOBase
but from which I cannot load a grib file at the moment since cfgrib
seems to only take file system paths.
Hello,
I agree this would be a useful feature. It's not there yet, but perhaps in the meantime you could be interested in trying our new package 'earthkit-data', which does enable this. In fact, it uses cfgrib in the background, so the xarray dataset will be the same.
Warning: earthkit-data is currently beta, and not yet for operational use, but it is available on PyPi and conda-forge. It has the same dependency on ecCodes that cfgrib has.
https://github.com/ecmwf/earthkit-data
https://earthkit-data.readthedocs.io/en/latest/index.html
Here's an example of how to create a bytes object (which you already have) and create an xarray from it. I should mention that this method does not create any temporary file (that would be cheating!).
import io
import earthkit.data
f = open('tests/data/t_time_series.grib', 'rb')
buffer = f.read()
f.close()
stream=io.BytesIO(buffer)
data = earthkit.data.from_source("stream", stream, batch_size=0)
x = data.to_xarray()
If you need to pass any kwargs to cfgrib, this shows how:
https://earthkit-data.readthedocs.io/en/latest/_api/data/readers/grib/index/index.html#data.readers.grib.index.FieldList.to_xarray
Best regards,
Iain
Kerchunk does this: https://github.com/fsspec/kerchunk/blob/main/kerchunk/grib2.py
import eccodes
import cfgrib
data # type: bytes
mid = eccodes.codes_new_from_message(data)
m = cfgrib.cfmessage.CfMessage(mid)