Giters

ecmwf / cfgrib

A Python interface to map GRIB files to the NetCDF Common Data Model following the CF Convention using ecCodes

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

ECCODES ERROR: unable to convert endStep in stepUnits

kevinrosa opened this issue · comments

commented

This may or may not be the right place to discuss this issue but it seemed related to some other previous discussions regarding time dimensions) and ECCODES ERROR message.

I'm working with WRF model GRIB2 output files. I receive the message ECCODES ERROR : unable to convert endStep in stepUnits but it does not actual raise an exception. And from what I can tell, the time variable that is returned in the xarray DataSet seems accurate.

  • I have 1 file per timestep, with 30 minutes between files. The ECCODES message appears only when reading the files on the half-hour, not the hour (i.e. *_30_d02.grb2).
  • The message is repeated 16 times per xr.open_dataset() call.
  • Opening with backend_kwargs={'time_dims': ['valid_time']} reduces this to 8 messages per open.
  • If I create an *.idx file, the ECCODES message is only returned on the first open, and not returned on subsequent opens once the index file has been created.
    • There is some nuance to this, however. If I open a file using filter_by_keys={'typeOfLevel': 'surface', 'stepType': 'instant'}, then I'll get the ECCODES message when trying to open the same file using {'typeOfLevel': 'heightAboveGround', 'level': 10}.
    • Then, if I try to open for a different height level e.g. {'typeOfLevel': 'heightAboveGround', 'level': 2}, I won't get the error.

The message originates from this line: https://github.com/ecmwf/eccodes/blob/develop/src/grib_accessor_class_g2end_step.cc#L293

Example usage:

fname = 'wrf_wrfout_2022-07-02.00.00-0024_30_d02.grb2'
ds = xr.open_dataset(fname, engine="cfgrib", filter_by_keys={'typeOfLevel': 'heightAboveGround', 'level': 2}, backend_kwargs={'indexpath': '', 'time_dims': ['valid_time']})

Example data file: wrf_wrfout_2022-07-02.00.00-0024_30_d02.grb2.zip
(Note, I zipped it since .grb2 files are not supported by github file attachments.

I've spend some time trying to figure this out and would be interested in any possible explanations or suggestions. At this point, I'd be happy just to convince myself that the resulting DataSet is correct and then figure out how to suppress the ECCODES message -- I've tried setting logging levels and catching warnings, but nothing has worked yet.

commented

So based on @shahramn reference I take it that WRF is incorrectly using a reserved attribute type?
There is a great deal of existing operational WRF output from NOAA which generates these errors
gs://high-resolution-rapid-refresh/hrrr.20230928/conus/hrrr.t00z.wrfsubhf01.grib2
Can this be reconciled in eccodes?

commented

We are working on a feature to support sub-hourly GRIB2 data. This is still in early stages but in the future the output of grib_ls will look something like this:

% grib_ls -n time wrf_wrfout_2022-07-02.00.00-0024_30_d02.grb2
stepType      stepunits     startStep     endStep       stepRange
instant       m             1470m         1470m         1470m
...
accum         m             1440m         1470m         1440m-1470m

So for data encoded in minutes, we will add the step unit to the stepRange key so that this becomes, e.g., "0m-30m" for an accumulation from 0 to 30 minutes (it will, however, remain as e.g., "0-2" - so without the unit - for an accumulation from 0 to 2 hours).
We will keep you posted

commented

@shahramn is there a github issue I can follow or some other place to track progress?
Thank you for your work!

commented

Best thing to do is watch the ecCodes GitHub repo for new releases here:
https://github.com/ecmwf/eccodes/releases

Also you can watch the "History Of Changes" in the ecCodes home page:
https://confluence.ecmwf.int/display/ECC/History+of+Changes