S3 Grib support
tdrwenski opened this issue · comments
Currently, using GRIB files in an ObjectStore don't work correctly due to the index files (.gbx9) being assumed to be local files. No index file is created if the expected one is not found, since it tries to create a directory using the s3 path which fails.
Expected behavior:
- use or update ObjectStore index file if available
- use local cache (if that setting is turned on) for storing the index file
- create or update the index file in the ObjectStore next to the data files if the cache is not turned on
The first and last bullet require creating or updating objects in the ObjectStore, which can be handled through MFile
. We could also choose to start with only using a local cache for index files.
Another thing to consider is that (in my opinion) an S3 source of grib files will be from places like ECMWF or one of the Open Data buckets, which is read-only for end-users.