google / weather-tools

Tools to make weather data accessible and useful.

Home Page:https://weather-tools.readthedocs.io/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Add optional feature to group common hypercubes into image collections when loading grib2 data into Earth Engine.

alxmrs opened this issue · comments

There is a significant performance problem that occurs when trying to convert a large grib file into a single COG. It seems like pyramiding data in GDAL takes exponential time (probably a bug).

Thus, there's an advantage to splitting up large grib files into multiple ImageCollections / COGS. At the same time, we'd like to group COGs that come from the same source together, so it's easy to access groups of variables as bands. For example, we don't want to write one COG per band, as that would make the access pattern more difficult.

The solution that @fredzyda, @j9sh264 and I have discussed for this is to make one COG / ImageCollection per layer of the grib file. As described here, cfgrib will naturally return multiple datasets when opening a mixed hypercubes of data in a single grib. These offer natural groupings to break up the expensive COG operation.

Let's implement this capability generically, where we default to normalized grib schemas (#184), and let users opt-in to creating multiple image collections in Earth Engine.