cfgrib loses all sub-hourly data with newer eccodes version
donni-h opened this issue · comments
What happened?
We have a pipeline that periodically scrapes GRIB files from the DWD with timesteps of 15 minutes each.
I have a old sample dataset from October 2023 that works flawlessly, showing all timesteps (roughly 3.600.000 rows).
However, something in the metadata from the DWD must have changed, because the current GRIB files throw this error:
DatasetBuildError: multiple values for unique key, try re-open the file with one of: filter_by_keys={'stepUnits': 1} filter_by_keys={'stepUnits': 0}
If I end up reopening the file with EITHER stepUnits: 0 or 1 it only shows roughly 900.000 rows, meaning 3/4 of the data is lost.
I would say this is DWD's issue to sort out, however our server instance is still running an older version of eccodes (2.20.0) and it doesn't cause any issues there, only locally with a more recent eccodes version.
What are the steps to reproduce the bug?
brew install eccodes
Get any icon_d2-germany_regular-lat-lon-* file from here
We are using poetry for python dependency management and are using cfgrib version 0.9.10.4 both locally and on the server
xarray.open_dataset(path_to_file, engine='cfgrib')
Version
tested 0.9.10.4 and 0.9.11
Platform (OS and architecture)
MacOS Intel - Sonoma 14.4.1
Relevant log output
I have debugged the issue and noticed that there is a single difference in the index header values.
This is the value of an object during Debugging.
Old (working) file:
'step': [24.0, 24.25, 24.5, 24.75]
Current (broken) file:
'step': [24.0, 'undef']Accompanying data
No response
Organisation
No response
I am facing the same issue. steps is missing on all datasets with the latest cfgrib version.
This should be fixed with the newly-released cfgrib version 0.9.12.0. Please re-open this issue if you still experience this problem!
@iainrussell thanks for fixing and inform us about that.
With the latest eccodes version I still receive the following error:
ECCODES ERROR : Unable to get startStep as long (Key/value not found)
ECCODES ERROR : Unable to find accessor startStepUnit
2.35.0
Before I was on 2.30.2.
Does this belong to a similar issue? Would be great if you have an idea where to find for a solution.