PigeonKeeper

Installation

$ npm install pigeonkeeper

Quick Start

var PigeonKeeper = require("pigeonkeeper");
var GenericService = require("./node_modules/pigeonkeeper/lib/genericService");

var mySharedData = {a:1, b:2};

function finalCallback(err, data)
{
    console.log("Current PK version: " + pk.getVersion());

    if(err)
    {
        console.log("In app.finalCallback, error object: " + JSON.stringify(err));
        console.log(pk.overallStateAsString());
        console.log(pk.getResults());
    }
    else
    {
        console.log("In app.finalCallback: all tasks are complete!");
        console.log(pk.overallStateAsString());
        console.log(pk.getResults());
    }
}

// Create generic services...
var svc7 = new GenericService("7");
var svc5 = new GenericService("5");
var svc3 = new GenericService("3");
var svc11 = new GenericService("11");
var svc8 = new GenericService("8");
var svc2 = new GenericService("2");
var svc9 = new GenericService("9");
var svc10 = new GenericService("10");


var pk = new PigeonKeeper("MyPigeonKeeper", finalCallback, false, 5);

// The digraph we're about to create matches the example given at http://en.wikipedia.org/wiki/Topological_sort
// Create vertices and associate processes
pk.addVertex("7", svc7, svc7.doStuff);
pk.addVertex("5", svc5, svc5.doStuff);
pk.addVertex("3", svc3, svc3.doStuff);
pk.addVertex("11", svc11, svc11.doStuff);
pk.addVertex("8", svc8, svc8.doStuff);
pk.addVertex("2", svc2, svc2.doStuff);
pk.addVertex("9", svc9, svc9.doStuff);
pk.addVertex("10", svc10, svc10.doStuff);

// Now add edges
pk.addEdge("7", "11");
pk.addEdge("7", "8");
pk.addEdge("5", "11");
pk.addEdge("3", "8");
pk.addEdge("3", "10");
pk.addEdge("11", "2");
pk.addEdge("11", "9");
pk.addEdge("11", "10");
pk.addEdge("8", "9");

// Kick it!
pk.start(mySharedData);

Introduction

What is PK?

A JS object for orchestrating the execution of processes or services

What PK is NOT

A workflow system
A visual programming language

Concepts

Digraphs

Short for "directed graphs"
Consists of vertices and directed edges

Vertices

Each vertex has…

An ID
A data object
A state (more about this later)

Directed Edges

Think: an arrow between two vertices
Specified by a start vertex and an end vertex
Parallel edges are NOT allowed

Directed Acyclic Graph

A digraph without cycles or self-loops
Abbreviated DAG

From: http://en.wikipedia.org/wiki/Topological_sort

Topological Sort

An ordering of the vertices so that any vertex appears after its parents
Not possible in digraphs with cycles (hence our focus on DAGs)
Topological ordering need not be unique!

Example

Some topological orderings of this DAG include:

7, 5, 3, 11, 8, 2, 9, 10 (visual left-to-right, top-to-bottom)
3, 5, 7, 8, 11, 2, 9, 10 (smallest-numbered available vertex first)
3, 7, 8, 5, 11, 10, 2, 9 (because we can)
5, 7, 3, 8, 11, 10, 9, 2 (fewest edges first)
7, 5, 11, 3, 10, 8, 9, 2 (largest-numbered available vertex first)
7, 5, 11, 2, 3, 8, 9, 10 (attempting top-to-bottom, left-to-right)

From: http://en.wikipedia.org/wiki/Topological_sort

Kahn's Algorithm

Used by PK for performing topological sort
Run time is O(|V| + |E|) where:
```
|V| is number of vertices
```
```
|E| is number of directed edges
```
Operates by removing parentless vertices (think: onion peeling)
Finite acyclic digraphs must always have at least one parentless vertex, if |V| > 1
Detects cycles!
Is "destructive," so digraph must be deep-copied

Processes

A process is a JS EventEmitter
Things that can happen inside a process:

HTTP requests, e.g. getting data from SOR

```
MongoDB or Solr/Lucene interactions
```
```
File operations
```
```
Synchronous operations, too
```
PK is agnostic about how the results of the process are stored

More About Processes

Processes have a "start" or "run" or "doStuff" method
As it goes, PK will call that method...
...and will pass in a “SharedData” object
Processes emit two types of events: ** "success" – includes data generated by the process ** "error" – includes error object

Processes and Vertices

We associate a vertex with a process
Vertex tells the process when to start running
Process updates the vertex when the process completes

Processes and Edges

Process B depends on Process A means...�A must be completed in order for B to start
Example: B uses data fetched by A
Since we associate processes with vertices, we describe dependencies between the processes with directed edges!

SharedData

SharedData is an object
Intended to store config info...
...but can be used however the developers want
Passed to PK when PK is started

Processes and Shared Data

PK passes (by reference) the SharedData object to each process when PK runs it
Process can modify that object…
```
When process starts, or...
```
```
When process ends successfully, or...
```
```
When process ends in failure
```
Idea is that processes can send config info around from one process to another

Again, What is PigeonKeeper? PigeonKeeper...

Is a JS object for orchestrating the execution of processes
Maintains association between vertices and processes
Maintains dependencies between processes
Performs topological sort
Maintains a SharedData object
Applies state transition rules (more in a sec)
Runs processes based on state of associated vertices

Available States

Each vertex can be in one of 5 states:

NOT_READY
READY
IN_PROGRESS
SUCCESS
FAIL

Distinction between READY and IN_PROGRESS must be maintained if we wish to limit number of processes running in parallel.

Lifecycle of a Vertex State

Transition Rules

NOT_READY → READY when either...

the vertex has no parents,
or all parents are in SUCCESS state

READY → IN_PROGRESS

controlled by PigeonKeeper

IN_PROGRESS → SUCCESS or FAIL

depends on the process associated with the vertex

FAIL propagates to all children

Example State Transitions

####Step 0####

Initially, all states are NOT_READY

####Step 1####

States without parents become READY

####Step 2####

Processes for the READY states are executed, and so states become IN_PROGRESS

####Step 3####

Some processes return OK, others fail; corresponding vertices’ states become SUCCESS or FAIL

####Step 4####

Vertices with SUCCESSful parents become READY; vertices with FAILed parents also FAIL

####Step 5####

READY vertices become IN_PROGRESS; FAILure propagates to children (sort of like the welfare state)

####Step 6####

Process associated with vertex 11 is SUCCESSful...

####Step 7####

...Child vertex 2 becomes READY...

####Step 8####

...then IN_PROGRESS...

####Step 9####

...and finishes SUCCESSfully.

About FAIL...

PigeonKeeper has two ways of handling failure – specified in constructor:

Entire PigeonKeeper stops when one vertex FAILs, or...
Vertices which depend on a FAILed process are also marked as FAIL, but other processes can continue

Above example used option 2!

Usage Steps

Overview

Write processes as event emitters
Write “finalProcess” method
Create a PK instance
Add vertices and associated processes
Add directed edges
Start PK!

Important Methods

Constructor

PigeonKeeper(pkName, finalCallback, quitOnFailure, maxNumRunningProcesses, logger, userObject)

pkName: instance name, which will be modified into a GUID
finalCallback: function to be called when PK quits
quitOnFailure: Boolean

When true, PK quits when a single process fails

When false, PK tries to execute all processes that don’t depend on failed processes

maxNumRunningProcesses
logger: a logging mechanism (optional)
userObject: object for use by logging mechanism (optional)

Commonly Used Methods

To create the digraph and associate processes with vertices, use...

addVertex(vertexId, service, serviceStart)

addEdge(startVertexId, endVertexId)

To start PK a'runnin', use...

start(sharedData)

Kicks-off the PK (begins applying state transition rules, starts processes, etc)
sharedData is passed to processes when they are ran
sharedData is intended to make config info available to the processes
Processes can modify sharedData

License

The MIT License (MIT)

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

montagist / PigeonKeeper