google / weather-tools

Tools to make weather data accessible and useful.

Home Page:https://weather-tools.readthedocs.io/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

`weather-sp`: Support arbitrary hypercube splits

alxmrs opened this issue · comments

Our initial use case of weather-sp was to split monolithic files by variable. Now, we've encountered a case where we'd like to split atmospheric data by model level.

To better generalize to all future uses of the splitter, let's re-design the tool to support arbitrary divisions of the data hypercube. This, ultimately, would help us improve usage of the weather-dl tool: If we didn't have to worry about the final file structure (because we could make it uniform), we could better optimize downloads. Specifically, we can get as much data as we can under the memory limit, which is ~75 GB for MARS as of writing.

#147 and #125 fix this