abhagsain/redis-smq

A simple high-performance Redis message queue for Node.js.

RedisSMQ is a Node.js library for queuing messages (aka jobs) and processing them asynchronously with consumers. Backed by Redis, it allows scaling up your application with ease of use.

For more details about initial RedisSMQ design and the motivation behind it see https://medium.com/@weyoss/building-a-simple-message-queue-using-redis-server-and-node-js-964eda240a2a

Current MQ Architecture Overview

High-level overview of how RedisSMQ works:

An application publishes messages using a producer.
Consumers pull messages off queues and start processing.
If an error occurs, messages are unacknowledged. Otherwise, once acknowledged, messages are moved to the acknowledged queue.
Unacknowledged messages are re-queued with optional retryDelay. When retryThreshold is exceeded, messages are put in the deal-letter queue.

Features

High-performance message processing
Scalable: A queue can be consumed by multiple concurrent consumers, running on different hosts.
Persistent: Messages are not lost in case of consumer failures.
Atomic: A message can be delivered only to one consumer at a time.
Message expiration: A message will not be delivered if it has been in a queue for longer than a given amount of time, called TTL (time-to-live).
Message consume timeout: Timeout for a consumer to consume a message.
Delaying and scheduling message delivery: Messages can be configured to be delayed, delivered for N times with an optional period between deliveries, and to be scheduled using CRON expressions.
Reliable Priority Queues: Supports priority messaging.
HTTP API: an HTTP interface is provided to interact with the MQ.
Web UI: Using the Web UI you can monitor and manage the MQ is real-time.
JSON Logging: Supports JSON log format for troubleshooting and debugging.
Highly optimized: Implemented using pure callbacks, with small memory footprint and no memory leaks. See callbacks vs promises vs async/await benchmarks.
Configurable: Many options and features can be configured.
Rigorously tested: With 79+ tests and code coverage no less than 80%.
Supports both redis & ioredis: RedisSMQ can be configured to use either redis or ioredis to connect to Redis server.

Table of content

What's new?

2021.11.22

Starting with RedisSMQ v5, you can now manage your queues and messages from the Web UI. Also, many changes and improvements has been made, allowing for better user experience and system stability. If you are upgrading your installation, take a look at the migration guide before proceeding.

2021.11.02

v4 is out with significant performance improvements and new features including the ability to fetch/delete/requeue messages from different queues using the MessageManager/QueueManager or with the help of the HTTP API.

See CHANGELOG for more details.

Installation

npm install redis-smq --save

Considerations:

RedisSMQ is targeted to be used in production environments. Therefore, only active LTS and maintenance LTS Node.js releases (v12, v14, and v16) are supported. The latest stable Node.js version is recommended.
Minimal Redis server version is 2.6.12. The latest stable Redis version is recommended.

Configuration

See Configuration for more details.

Usage

Basics

RedisSMQ provides 3 classes: Message, Producer, and Consumer in order to work with the message queue.

Message Class

Message class is responsible for creating and manipulating messages.

const { Message } = require('redis-smq');
const message = new Message();
message
    .setBody({hello: 'world'})
    .setTTL(3600000); // in millis

let messageTTL = message.getTTL();

See Message Reference for more details.

Producer Class

Producer class is in turn responsible for publishing messages. Each Producer instance is associated with a message queue and provides the produceMessage() method to publish a message.

// filename: ./examples/javascript/ns1-test-queue-producer.js

'use strict';
const { Message, Producer } = require('redis-smq');

const message = new Message();

message
    .setBody({hello: 'world'})
    .setTTL(3600000);

const producer = new Producer('test_queue');
producer.produceMessage(message, (err) => {
   if (err) console.log(err);
   else console.log('Successfully produced')
});

See Producer Reference for more details.

Consumer Class

The Consumer class is the parent class for all your consumers, which are required to implement the abstract method consume() from the parent class.

Once a message is received, the consume() method get invoked with the received message as its first argument.

In a typical scenario, consumers are saved per files, so that each file represents a consumer, which can be started from CLI as shown in the example bellow.

// filename: ./examples/javascript/ns1-test-queue-consumer.js
'use strict';

const { Consumer } = require('redis-smq');

class TestQueueConsumer extends Consumer {
    consume(message, cb) {
        console.log('Got a message to consume:', message);
        cb();
    }
}

const consumer = new TestQueueConsumer('test_queue');
consumer.run();

Starting a consumer:

$ node ./examples/javascript/test-queue-consumer.js

To acknowledge a received message, you invoke the callback function without arguments.

The message acknowledgment informs the MQ that the message has been successfully consumed.

If an error occurred, the message is unacknowledged by passing the error to the callback function.

By default, unacknowledged messages are re-queued and delivered again unless message retry threshold is exceeded. Then the messages are moved to dead-letter queue (DLQ).

Each message queue has a system generated corresponding queue called dead-letter queue that holds all messages that couldn't be processed or can not be delivered to consumers.

See Consumer Reference for more details.

Advanced Topics

Performance

See Performance for more details.

Contributing

So you are interested in contributing to this project? Please see CONTRIBUTING.md.

License

MIT

abhagsain / redis-smq