littleflow
A little flow language with simple objectives:
- an expression language for defining the flow of tasks
- a core algorithm for invocation of workflows and tasks
- flows are independent of task implementation or workflow deployments
The littleflow project has an independent flow language, a workflow engine, and provides integrations with Redis and Kubernetes.
A quick introduction
Tasks are simply referenced by name:
A
The can also have parameters in the form of JSON or YAML literals:
A(- hello: world -)
You can sequence tasks together with the arrow operator (i.e., → U+2192 or ->) into flow statements:
A → B → C
And the set of flow statements define a workflow graph:
A → B :x → C;
:x → D
stateDiagram-v2
direction LR
state "A" as A.1
state "B" as B.2
state "C" as C.3
state "D" as D.4
[*]-->A.1
A.1-->B.2
B.2-->C.3
B.2-->D.4
C.3-->[*]
D.4-->[*]
You can run workflows programmatically in python (see example-run.py):
from littleflow import run_workflow
workflow = """
A → B :x → C;
:x → D
"""
def A(input):
print('A')
def B(input):
print('B')
def C(input):
print('C')
def D(input):
print('D')
run_workflow(workflow,locals())Or compile and run workflows for more complex interactions:
from littleflow import Parser, Compiler, Context, Runner
workflow = """
A → B :x → C;
:x → D
"""
p = Parser()
c = Compiler()
model = p.parse(workflow)
flow = c.compile(model)
context = Context(flow)
runner = Runner()
runner.start(context)
while not context.ending.empty():
runner.next(context,context.ending.get())There is a language specification for the workflow expressions.
Also, the Redis and Kubernetes integrations provide the ability to execute workflows in distributed context for remote execution of tasks and for long-running workflows. Further, the whole system can be deployed on Kubernetes.
Getting started
You can pip install the library:
pip install littleflow
Workflows are specify in a language called littleflow. This workflow language allows you to describe the flow of steps as a graph and various parameters.
A workflow is compiled into a graph that can be executed asynchronously. The workflow itself is represented by a matrix and an index of the tasks. The state of the workflow is stored in a few vectors. This allows a simple stateless algorithm run the workflow forward from any state represented by those vectors. As such, those vectors can be save and restored from storage.
The core library provides:
- the language parser and compiler
- the core algorithm
- base classes for building more complex execution environments (e.g., for distributed execution or state storage)
There is also:
- A Redis integration for caching workflow state information and workflow events to enable distrubuted execution
- A Kubernetes integration that builds on the Redis integration to provide a way to execute tasks as Kubernetes jobs.
- A Kubernetes deployment that provides a way to deploy the framework on your Kubernetes cluster via kustomize.
Commands
The littleflow module is executable and provide several commands.
compile
python -m littleflow compile WORKFLOW
where WORKFLOW is a littleflow workflow file.
This command will parse and compile a workflow and output various information about the workflow.
doc
python -m littleflow doc WORKFLOW
where WORKFLOW is a littleflow workflow file.
This command will generate a mermaid diagram that represents the workflow graph.
run
python -m littleflow run WORKFLOW
where WORKFLOW is a littleflow workflow file.
This command will run a workflow with a pseudo-task executor that runs each task as a simple echo of input to output.
This command is primarily for testing purposes. The integration with redis provides a way to run workflows with real task executors.
version
python -m littleflow version
This command outputs the littleflow version.