Nightly is a system for scheduling, monitoring and collecting outputs from configuration-defined command-line tasks. It is developed to support my daily research work since I can't find an existing solution that is lightweight (depends on nothing more than python, mysql and ssh) and customizable (written in less than a few hundreds lines of Python in total). I hope it might be useful to you as well.
Features:
- Define compute/storage resources declaratively
- Define job declaratively (similar to CI)
- Run job periodically, or as a daemon, or just once
- For "just once", you can also run an ad-hoc command line job without defining it
- Automatically transfer job log and output to storage
- Monitor the progress via web panel
- Receive notification in job success/fail/...
Data objects:
- Job
- Resource
Components:
- Log DB: database containing job log (status) entries
- Scheduler: master node, launch jobs and monitor process status
- Runner: launch job on runner node through SSH
- Storage: persisting job output and selected files
- Panel: web panel daemon
- Notif: notification (only email for now)
poetry shell # get poetry: https://python-poetry.org/
poetry install
# symblink the virtual env created by poetry to `.venv` under project root as well
There should be two files inside config
folder, which is ignored by git.
See example_config
for examples and comments.
Users can use different config
folders in different environment (production or development).
See https://app.sendgrid.com/guide/integrate/langs/python for email notification configuration keys
sendgrid_api_key
, notif_sender_email
and notif_receiver_email
.
Initialize or update database:
make upgrade-db
Truncate and backup log table:
make truncate-all-log
Run the master daemon for scheduling jobs (on a server within tmux):
make run # it will tee the log to main.log as well
Run web panel daemon:
make panel # debug mode
export BASIC_AUTH_USERNAME=...; export BASIC_AUTH_PASSWORD=...; make panel-prod
Currently, panel server and master daemon must be on the same node.
Launch an ad-hoc job from command line:
scripts/launch-nightly sleep 10 # launch an ad-hoc one on the master node
scripts/launch-nightly --job ... # launch a defined job. Note that it must has "once" schedule
Homepage:
Output page:
Email notification: