jiajunhuang / toq

task queue for golang

toq

toq pronounces to queue, which is a distributed task queue.

Features

• Concurrent task executor
• Easy to learn and use
• Language non-specific protocol
• Distributed(with support by broker, like Redis)
• Retry
• Graceful restart
• Redis as broker
• Exception/Panic handle
• Retry countdown
• RabbitMQ as broker
• Kafka as broker
• NSQ as broker
• HA

Usage

toq is a realy simple task queue, if you want to implement a worker, you just have to define a function whose type is:

type Worker func(task.Task) task.Result

And register it with a key to consumer engine(of course you have to initialize a consumer engine first), and call c.Dequeue(), then everything is just ok, consumer engine will auto discover tasks and match the specify worker by the key you given. just like this:

package main

import (
	"flag"
	"time"

	"github.com/jiajunhuang/toq/consumer"
	"github.com/jiajunhuang/toq/task"
	"github.com/sirupsen/logrus"
)

var (
	redisPasswd = flag.String("redisPasswd", "", "")
	redisURI    = flag.String("redisURI", "", "")
	redisDBNum  = flag.Int("redisDBNum", 0, "")
	maxIdle     = flag.Int("maxIdle", 1024, "")
	maxActive   = flag.Int("maxActive", 100, "")
)

func Run(t task.Task) task.Result {
	logrus.Infof("running task %s...", t.ID)
	time.Sleep(10 * time.Second)
	logrus.Infof("task %s succeed", t.ID)

	return task.Result{TaskID: t.ID, State: task.ResultStateSucceed, Message: "succeed"}
}

func main() {
	flag.Parse()

	redisPool := NewRedisPool()
	c := consumer.NewConsumer(redisPool, []string{"test_toq_queue"})
	if err := c.RegisterWorker("test_key", Run); err != nil {
		logrus.Errorf("failed to register worker with error: %s", err)
	}
	c.Dequeue()
}

Really simple, right?

for now, toq just support redis as broker, but it will support more in the future.

Producer

If you want to produce tasks, you just have to do something like this:

package main

import (
	"flag"

	"github.com/jiajunhuang/toq/producer"
	"github.com/jiajunhuang/toq/task"
	"github.com/sirupsen/logrus"
)

func main() {
	flag.Parse()

	redisPool := NewRedisPool()
	p := producer.NewProducer(redisPool)
	for {
		taskID := UUID4()
		logrus.Infof("enqueue task %s", taskID)
		t := task.Task{ID: taskID, Key: "test_key", Args: "{}"}
		p.Enqueue("test_toq_queue", t)
	}
}

If you want to learn more, please visit https://github.com/jiajunhuang/toq