Goblero

Pure Go, Simple, Embedded, Persistent Job Queue, backed by BadgerDB

DO NOT USE IN PRODUCTION This library is still in Alpha / Work In Progress

About Goblero

Intro article: Introducing Goblero: a Go Embedded Job Queue

Pure Go library, no cgo
Simple, embedded, persistent job queue
Provides in-process job processing to any Go app
The jobs/status changes are persisted to disk after each operation and pending jobs can continue processing after an app restart or a crash
Allows multiple "processors", each processor/worker processes one job at a time then is assigned a new job, etc
The storage engine used is BadgerDB

P.S: Why is the library named Goblero ? Go for the Go programming language obviously, and Badger in french is "Blaireau", but blero is easier to pronounce :)

Usage

The full API is documented on godoc.org. There is also a demo repo goblero-demo

Get package

go get -u github.com/didil/goblero/pkg/blero

API

// Create a new Blero backend
bl := blero.New("db/")

// Start Blero
bl.Start()

// defer Stopping Blero
defer bl.Stop()

// register a processor
bl.RegisterProcessorFunc(func(j *blero.Job) error {
  // Do some processing, access job name with j.Name, job data with j.Data
})

// enqueue a job
bl.EnqueueJob("MyJob", []byte("My Job Data"))

Benchmarks

# Core i5 laptop / 8GB Ram / SSD 
make bench
BenchmarkEnqueue/EnqueueJob-4          50000            159942 ns/op (~ 6250 ops/s)
BenchmarkEnqueue/dequeueJob-4           5000           2767260 ns/op (~  361 ops/s)

Todo:

Restart interrupted jobs after app restart/crashes
Sweep completed jobs from the "complete" queue
Failed Jobs retry options
Allow batch enqueuing
Add support for Go contexts
Test in real conditions under high load
Expose Prometheus Metrics in an Http handler
Optimize performance / Locking

Contributing

All contributions (PR, feedback, bug reports, ideas, etc.) are welcome !

didil / goblero