Qu

Qu is a Ruby library for queuing and processing background jobs. It is heavily inspired by delayed_job and Resque.

Qu was created to overcome some shortcomings in the existing queuing libraries that we experienced at Ordered List while building SpeakerDeck, Gaug.es and Harmony. The advantages of Qu are:

Multiple backends (redis, mongo)
Jobs are requeued when worker is killed
Resque-like API

Information & Help

Find more information on the Wiki.
Post to the Google Group for help or questions.
See the issue tracker for known issues or to report an issue.

Installation

Rails 3 and 4

Decide which backend you want to use and add the gem to your Gemfile.

gem 'qu-rails'
gem 'qu-redis'

That's all you need to do!

Rails 2

Decide which backend you want to use and add the gem to config.gems in environment.rb:

config.gem 'qu-redis'

To load the rake tasks, add the following to your Rakefile:

require 'qu/tasks'

Usage

Jobs are defined by extending the Qu::Job class:

class ProcessPresentation < Qu::Job
  def initialize(presentation_id)
    @presentation_id = presentation_id
  end

  def perform
    Presentation.find(@presentation_id).process!
  end
end

You can add a job to the queue by calling create on your job:

job = ProcessPresentation.create(@presentation.id)
puts "Created job #{job.id}"

The job will be initialized with any parameters that are passed to it when it is performed. These parameters will be stored in the backend, so they must be simple types that can easily be serialized and unserialized. Don't try to pass in an ActiveRecord object.

Processing the jobs on the queue can be done with a Rake task:

$ bundle exec rake qu:work

You can easily inspect the queue or clear it:

puts "Jobs on the queue:", Qu.size
Qu.clear

Queues

The default queue is used, um…by default. Jobs that don't specify a queue will be placed in that queue, and workers that don't specify a queue will work on that queue.

However, if you have some background jobs that are more or less important, or some that take longer than others, you may want to consider using multiple queues. You can have workers dedicated to specific queues, or simply tell all your workers to work on the most important queue first.

Jobs can be placed in a specific queue by setting the queue:

class CallThePresident < Qu::Job
  queue :urgent

  def initialize(message)
    @message = message
  end

  def perform
    # …
  end
end

You can then tell workers to work on this queue by passing an environment variable

$ bundle exec rake qu:work QUEUES=urgent,default

Note that if you still want your worker to process the default queue, you must specify it. Queues will be process in the order they are specified.

You can also get the size or clear a specific queue:

Qu.size(:urgent)
Qu.clear(:urgent)

Configuration

Most of the configuration for Qu should be automatic. It will also automatically detect ENV variables from Heroku for backend connections, so you shouldn't need to do anything to configure the backend.

However, if you do need to customize it, you can by calling the Qu.configure:

Qu.configure do |c|
  c.logger = Logger.new('log/qu.log')
  c.graceful_shutdown = true
end

Tests

If you prefer to have jobs processed immediatly in your tests, there is an Immediate backend that will perform the job instead of enqueuing it. In your test helper, require qu-immediate:

require 'qu-immediate'

Why another queuing library?

Resque and delayed_job are both great, but both of them have shortcomings that can be frustrating in production applications.

delayed_job was a brilliantly simple pioneer in the world of database-backed queues. While most asynchronous queuing systems were tending toward overly complex, it made use of your existing database and just worked. But there are a few flaws:

Occasionally fails silently.
Use of priority instead of separate named queues.
Contention in the ActiveRecord backend with multiple workers. Occasionally the same job gets performed by multiple workers.

Resque, the wiser relative of delayed_job, fixes most of those issues. But in doing so, it forces some of its beliefs on you, and sometimes those beliefs just don't make sense for your environment. Here are some of the flaws of Resque:

Redis is a great queue backend, but it doesn't make sense for every environment.
Forking before each job prevents memory leaks, but it is terribly inefficient in environments with a lot of fast jobs (the resque-jobs-per-fork plugin alleviates this)

Those shortcomings lead us to write Qu. It is not perfect, but we hope to overcome the issues we faced with other queuing libraries.

Contributing

If you find what looks like a bug:

Search the mailing list to see if anyone else had the same issue.
Check the GitHub issue tracker to see if anyone else has reported issue.
If you don't see anything, create an issue with information on how to reproduce it.

If you want to contribute an enhancement or a fix:

Fork the project on GitHub.
Make your changes with tests.
Commit the changes without making changes to the Rakefile, Gemfile, gemspec, or any other files that aren't related to your enhancement or fix
Send a pull request.

grantr / qu

Qu