Unidata / netcdf-java

The Unidata netcdf-java library

Home Page:https://docs.unidata.ucar.edu/netcdf-java/current/userguide/index.html

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

S3 Grib support

tdrwenski opened this issue · comments

Currently, using GRIB files in an ObjectStore don't work correctly due to the index files (.gbx9) being assumed to be local files. No index file is created if the expected one is not found, since it tries to create a directory using the s3 path which fails.

Expected behavior:

  • use or update ObjectStore index file if available
  • use local cache (if that setting is turned on) for storing the index file
  • create or update the index file in the ObjectStore next to the data files if the cache is not turned on

The first and last bullet require creating or updating objects in the ObjectStore, which can be handled through MFile. We could also choose to start with only using a local cache for index files.

Another thing to consider is that (in my opinion) an S3 source of grib files will be from places like ECMWF or one of the Open Data buckets, which is read-only for end-users.