NodeJS Background Jobs With Redis And Kue

Often when implementing your server, you may want to execute a task that takes some time (greater about 800ms). If a task is time intensive, it is usually a bad user experience to wait for the task to be completed before getting a response from the server. You do not want your front end waiting for minutes or longer with no visual indication of what is going on. A common way to solve this problem is to create a background job on the server.

Background jobs

The purpose of a background job is to do some longer running task on the server asynchronously. Since the task is asynchronous, the http response that the client is waiting for, does not depend on the result of your background job.

Email is a great example of a task that makes sense as a background job. The client may sign up with your server to create a user account. If the server plans to send a confirmation email, there is no need for the user to wait for the server to complete the task. The server can send a response once the user has been created in the database and a background job has been created to send the email asynchronously.

Common Technology for Creating Background Jobs

NodeJS

kue which requires redis -- PREFERRED
node-background-task also uses redis

Ruby on Rails

Sidekiq also uses redis -- PREFERRED
Resque also uses redis
Delayed Job

Redis

Since most of the above technologies use redis, it is a good idea to get familiar with it before moving on.

Redis is an in memory data storage system. What does in memory mean? Basically that the data you are storing in redis is very fast to access because it is all in your ram rather than stored on the hard drive. The downside of storing all the data in memory is that if the computer shuts down unexpectedly, you can lose all of your data. Therefore, redis is not great for saving mission critical information (use something like postgres for very important data), but it's great for saving short lived data that we are less concerned about losing.

Redis has a fairly large set of data types that you can store your data in. The full list of commands can be found here.

Redis Setup

Install:

brew install redis

Running:

redis-server

Interacting with redis server from command line:

redis-cli

EXERCISE

Use the redis-cli to interact with redis. Try different commands and see what they do. Refer to the redis commands in the docs. What do the following commands do? GET, SET, KEYS *, LPUSH, RPUSH, TYPE, etc.
Describe what Redis is in your own words. How is it different from Postgres or Mongo?

Kue

As you saw above, there are many possible technologies and servers to use when implementing background jobs. We are going to focus on kue for a few reasons:

Full feature set
Has a very easy to setup UI.
Kue is a npm package for NodeJS (Node is what we've been focusing on).

appSimple.js

To get started, look at appSimple.js. To get it running locally, do the normal setup:

npm install

Next, you must make sure redis is started in order to use kue. In a separate terminal window, type:

redis-server

and allow the server to run.

Next, run the app with

node appSimple.js

There are 4 main parts to using Kue in the simple example:

Make sure redis is running
Requiring kue and creating a queue.

var kue = require('kue');
var queue = kue.createQueue();

Creating a job to be placed in the queue. Notice that the first argument to create is a string called test. The name is important. It must match the name used to process the queue:

var job = queue.create('test', {
      title: title
  }).save(function(err) {
     if(err) { console.log("error: ", err); }
  });

Process the queue (Note the name of the queue to process is also test):

queue.process('test', 2, function(job, done){
  console.log(job.data.title);
  setTimeout(function() {done()}, 600);
});

Another cool thing to note about this example (but not required for it to work), is the kue app. The following line mounts the kue app at the path /kue

app.use('/kue', kue.app);

If you go to http://localhost:3000/kue, you should be able to see what is happening with your kue queue.

appEmail.js

A little more complicated example (because it actually does something) is in the appEmail.js file. The file has the same 3 setps:

Make sure redis is running
Require kue and create a queue.
Create background tasks to be saved to the queue.
Process the queue.

One thing to note in the code referenced below is that the task creation takes more parameters than in the first example. In fact, you can pass any object you want to the queue. It is generally a good idea to keep the object as simple as possible though. Also, this task will retry 3 times before it gives up. Since there are many reasons why email can fail and it is often not too harmful to try again, retries make sense.

  queue.create('email', {
    title: 'Get rich quick email scheme',
    subject: 'Get rich quick email scheme',
    to: 'elie.schoppik@galvanize.com',
    from: from,
    body: body
  }).attempts(3).save();

Also, notice how the queue processing works:

queue.process('email', function(job, done){
  email(job.data, done);
});

function email(emailData, done) {
  var mailOptions = {
    from: emailData.from,
    to: emailData.to,
    subject: emailData.subject,
    text: emailData.body,
    html: emailData.body
  };

  transporter.sendMail(mailOptions, function(error, info){
    if(error) {
      console.log(error);
      return done(error);
    }
    console.log('Message sent: ' + info.response);
    done();
    
  });
}

The processing simply calls an email function that does the work. Since the process method passes all the data to the email function, the email function can get access to all of the parameters defined when the task was created.