google / weather-tools

Tools to make weather data accessible and useful.

Home Page:https://weather-tools.readthedocs.io/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Absence of partition_keys in config file should not cause an exception

CillianFn opened this issue · comments

As discussed , it should be possible to not specify a partition key and download all data as one chunk

Currently the absence of partition_keys in the config file causes a KeyError

I am willing to work on this issue. please guild me further

Sure thing, @pranay101. Sometimes, users may want to download data in one chunk instead of smaller shards. This should be possible today, but isn't. Ideally, users should be able to pass a target template with no string templates, and thus no need to write partition_keys at all. The work for this issue involves update weather-dl's parser (around these lines) such that:

  1. users can create a target path with no templates
  2. if (1), then users do not need to specify any partition key argument
  3. update error messages to guide the user to the right config, given these changes
  4. update documentation related to these parser changes
  5. update and add to unit tests to check this behavior

Further, we need to test that the rest of the pipeline works e2e when there is only one shard worth of work. To my knowledge, this should definitely be the case, but it's always worth while to manually test (this can be done with a local run).

Fixed with #90