Absence of partition_keys in config file should not cause an exception
CillianFn opened this issue · comments
As discussed , it should be possible to not specify a partition key and download all data as one chunk
Currently the absence of partition_keys
in the config file causes a KeyError
I am willing to work on this issue. please guild me further
Sure thing, @pranay101. Sometimes, users may want to download data in one chunk instead of smaller shards. This should be possible today, but isn't. Ideally, users should be able to pass a target template with no string templates, and thus no need to write partition_keys at all. The work for this issue involves update weather-dl's parser (around these lines) such that:
- users can create a target path with no templates
- if (1), then users do not need to specify any partition key argument
- update error messages to guide the user to the right config, given these changes
- update documentation related to these parser changes
- update and add to unit tests to check this behavior
Further, we need to test that the rest of the pipeline works e2e when there is only one shard worth of work. To my knowledge, this should definitely be the case, but it's always worth while to manually test (this can be done with a local run).
Fixed with #90