Piper
Hello, this is Piper.
Piper is your friend.
Piper is also a pipeline library for Go heavily influenced by whiskybadger's article and Netty.
Operators
Operators are the basic building blocks for the pipeline. These come in two flavors: regular and sink. The former takes an input and an output channel and can be used either as an middle or output stage. The latter is used to end a pipeline that doesn't have an output.
var myOp = piper.Operator(func(in <-chan interface{}, out chan<- interface{}){
// cool stuff
})
var mySinkOp = piper.SinkOperator(func(in <-chan interface{}){
// cool stuff but without an output
})
Operators are later wrapped in Stages to be added to the pipeline.
What is a Pipeline?
A pipeline is a list of Stages with an input channel and optionally an output.
How do i build one?
With a PipelineBuilder of course! The builder provides a way to add stages to the pipeline. The entry channel can be buffered or not.
// a sample pipeline with an output
pipeline, out := piper.NewBuilder().
AddLastBuffered(1000, anOperator).
Output(anotherOperator)
// this one does not output anything
pipeline := piper.NewBuilder().
AddLast(anOperator).
Sink(aSinkOperator)
All building functions come with its buffered flavor.
Default Sink
Piper also includes the default sink stage. It receives messages and discards them.
pipeline := piper.NewBuilder().
AddLast(anOperator).
DefaultSink()
Splitting
Splitting is a way to create more complex pipelines. It works as a sink stage that ends the initial pipeline and forwards the output to a list of pipelines.
pipeline := piper.NewBuilder().
Split(aPipeline, anotherPipeline, ...)
The forwaring is made sequentially, so this could cause starvation if one of the pipelines stalls.
Ok, i have my pipeline. How do i use it?
pipeline.Run() // This starts the pipeline, duh
// You should stop the pipeline when you are done with it
// multiple stops can cause a panic
pipline.Stop()
// if you want to wait for the pipeline to consume all remaining messages:
pipeline.Wait()
// or maybe you want to wait for other things as well
<-pipeline.Done()
A "fun" example
package main
import (
"fmt"
"time"
"github.com/sebisujar/piper"
)
func main() {
plusOperator := piper.Operator(func(in <-chan interface{}, out chan<- interface{}) {
for _n := range in {
n := _n.(int)
out <- n + 1
}
})
squareOperator := piper.Operator(func(in <-chan interface{}, out chan<- interface{}) {
for _n := range in {
n := _n.(int)
out <- n * n
}
time.Sleep(2 * time.Second)
})
idOperator := piper.Operator(func(in <-chan interface{}, out chan<- interface{}) {
for _n := range in {
out <- _n
}
})
idPipeline, idOut := piper.NewBuilder().Output(idOperator) // complex pipeline
squarePipeline, squareOut := piper.NewBuilder().Output(squareOperator) // much more complex pipeline
p := piper.NewBuilder().
AddLast(plusOperator).
Split(idPipeline, squarePipeline)
go func() {
for x := range idOut {
fmt.Println("id pipeline: ", x)
}
}()
go func() {
for x := range squareOut {
fmt.Println("square pipeline: ", x)
}
}()
p.Run()
in := p.In()
in <- 1
in <- 2
in <- 3
p.Stop()
p.Wait()
fmt.Println("bye")
}