Add option to use pudl nightly builds
grgmiller opened this issue · comments
When running the OGE pipeline, we currently use the most recent versioned release of PUDL available on Zenodo. However, pudl is now publishing nightly builds of their data here: https://github.com/catalyst-cooperative/pudl#nightly-data-builds. This could allow us to use fresher data for testing purposes, and if we ever wanted to be able to use monthly data files (like #212).
To implement, we might just want to add a command line option to allow the user to specify where to download the pudl data from.
So after talking to the folks at Catalyst, it sounds like in addition to running nightly builds, they will also be doing more frequent/automatic versioned releases in zenodo. However, these zenodo releases will also be paired to a persistent version of the data on amazon cloud storage.
So in terms of next steps, we may want to:
- Change the dependency to grab the data from the cloud storage instead of zenodo, and figure out a way to automatically notify us when a new version is released.
- Set up a development version of OGE that uses the nightly builds and monitors that for changes.
Addressed by #318