Lightflow - a lightweight, distributed workflow system
Lightflow is a Python 3.5+ library and command-line tool for executing workflows, composed of individual tasks, in a distributed fashion. It is based on Celery and provides task dependencies, data exchange between tasks and an intuitive description of workflows.
Dependencies
Python
Lightflow is written in Python 3 and requires Python 3.5 or higher.
Operating system
Lightflow is being developed and tested on Linux, with Debian and RedHat being the main platforms.
redis
The redis database is required by Lightflow as a communication broker between tasks. It is also used as the default broker for the Celery queuing system, but could be replaced with any other supported Celery broker.
MongoDB
Lightflow makes use of MongoDB for storing persistent data during a workflow run that can be accessed by all tasks.
Getting started
The following getting started guide assumes a redis database running on localhost and port 6379
as well as a MongoDB database running on localhost and port 27017.
Install Lightflow from PyPi:
pip install lightflow
Create a default configuration file and copy the provided example workflows to a local directory of your choice:
lightflow config default . lightflow config examples .
If you like, list all available example workflows:
lightflow workflow list
In order to execute a workflow, start a worker that consumes jobs from the workflow, dag and task queues.
Then start a workflow from the list of available examples. The following example starts the workflow simple:
lightflow worker start lightflow workflow start simple