Redis Keyspace Notifications with Node-Redis

This is a small project using Redis and Redis Keyspace Notifications. Keyspace notifications allow clients to subscribe to Pub/Sub channels in order to receive events affecting the Redis data set in some way. In this project, we'll use them to model a high score table in a multi-player game.

Imagine we're building a game in which players have to collect tokens, and where we measure who is winning by keeping track of how many tokens each player has. We'll model this in Redis as follows:

We'll use a Redis Set to record which tokens each player has at any given time. The tokens are named "a", "b", "c" etc and each player has to collect all of them. We'll store these sets in the Redis keyspace using key names of the form: tokens:username where username is the player's user name in our game.
We'll use a Redis Sorted Set to keep track of the high scores. We only need a single key for this, and we'll call that scoreboard.

What we want to do is use keyspace notifications to track updates to those tokens:username Set keys for us. Whenever something changes with one, we want a Node.js script that's subscribed to some Pub/Sub channels to receive a message so that it can update the scoreboard. With this architecture, our code that handles updating the Sets doesn't need to know anything about the scoreboard's implementation, or even that we're keeping score.

Here's what this project looks like when running... in the left hand terminal I'm manipulating the contents of Redis Sets - one for each player that's collecting tokens in the game. On the right, the Node.js application is reacting to the keyspace events that are raised, and is updating the high score table that we keep as a Redis Sorted Set.

Prerequisites

To try out this project yourself, you'll need:

Redis (install locally or use the supplied Docker Compose file).
Docker Desktop (if using the Docker Compose file).
git command line tools.
Node.js version 14.8+ or Python version 3.6+.

If you're not familiar with how pub/sub works in Redis, you may want to read the guide on redis.io.

Get the Code

To get the code, clone the repo to your machine:

$ git clone https://github.com/redis-developer/keyspace-notifications-node-redis.git
$ cd keyspace-notifications-node-redis

Configuring and Starting Redis

For performance reasons, keyspace notifications are off by default. If you're using the supplied Docker Compose file, this will start a Redis Stack container and pass it extra configuration settings that enable a subset of keyspace notifications that we need for our application. If you go this route, there's nothing to do here - just start Redis Stack:

$ docker-compose up -d

When you want to stop the container:

$ docker-compose down

Alternatively, you could also turn on keyspace notifications by starting Redis then using the CONFIG SET command in redis-cli or using the command line built into RedisInsight (if using Docker Compose, RedisInsight is available at http://localhost:8001). Let's enable keyspace events (K) for both Redis Set commands (s) and generic commands (g - we will want this to catch DEL commands):

127.0.0.1:6379> config set notify-keyspace-events Ksg

If you're using a locally installed Redis with a redis.conf file, enable the keyspace notifications we want to use by changing the following line in that file:

notify-keyspace-events ""

to read:

notify-keyspace-events "Ksg"

then restart your Redis server.

Whichever method you use, verify that your configuration changes are active using redis-cli or the command line in RedisInsight:

$ redis-cli
127.0.0.1:6379> config get notify-keyspace-events
1) "notify-keyspace-events"
2) "gsK"

See the Keyspace Notifications configuration docs for more information on configuring events.

Application Setup

Check out the language-specific instructions:

Generating Events

You won't see any output from the application until you make changes in Redis that generate the sorts of keyspace events that you enabled earlier, and which affect the portion of the keyspace that the application is listening for. We'll look at how that works later in this document, but for now let's try changing data in Redis in ways that generate appropriate events and watch the application react to them and update our high score table...

First, with the application running, connect to Redis using redis-cli or RedisInsight.

Let's assume that a few users playing our game have found some of the tokens, and we want to represent that by adding each token to a Redis Set whose key name has the user name in it:

$ redis-cli
127.0.0.1:6379> sadd tokens:simon a b c
(integer) 3
127.0.0.1:6379> sadd tokens:suze a d e f
(integer) 4
127.0.0.1:6379> sadd tokens:steve a b c f g
(integer) 5

Take a look at the terminal where the Node.js application is running. You should see that each time we ran the SADD command, an event was generated and the application created and updated a high score table (stored as a Redis Sorted Set in a key named scoreboard):

event >>> sadd on __keyspace@0__:tokens:simon
Set cardinality tokens:simon is 3
Scores:
[ { value: 'simon', score: 3 } ]
event >>> sadd on __keyspace@0__:tokens:suze
Set cardinality tokens:suze is 4
Scores:
[ { value: 'suze', score: 4 }, { value: 'simon', score: 3 } ]
event >>> sadd on __keyspace@0__:tokens:steve
Set cardinality tokens:steve is 5
Scores:
[
  { value: 'steve', score: 5 },
  { value: 'suze', score: 4 },
  { value: 'simon', score: 3 }
]

Let's remove a token from one user, and delete another entirely:

127.0.0.1:6379> srem tokens:steve c
(integer) 1
127.0.0.1:6379> del tokens:simon
(integer) 1

The output from the Node.js application looks like this:

event >>> srem on __keyspace@0__:tokens:steve
Set cardinality tokens:steve is 4
Scores:
[
  { value: 'suze', score: 4 },
  { value: 'steve', score: 4 },
  { value: 'simon', score: 3 }
]
event >>> del on __keyspace@0__:tokens:simon
Set cardinality tokens:simon is 0
Scores:
[ { value: 'suze', score: 4 }, { value: 'steve', score: 4 } ]

How does it Work?

We're gonna be using the Node.js implementation to describe how the application works. The same logic applies to all the examples, regardless of the language of choice.

All of the code for this application is contained in a single file: index.js.

The keyspace notification events are generated by Redis itself, so there's no code required to create these. Instead our application needs to listen for them and take appropriate action.

The PSUBSCRIBE command is one of the ways that we can listen for pub/sub events. The P in PSUBSCRIBE stands for "Pattern". Redis publishes pub/sub events to named channels. This command listens for events on pub/sub channels that match a glob style pattern.

The PSUBSCRIBE command is also a blocking command. This means that the Redis client connection that we use it from becomes tied up for this purpose only and can't be used to send other commands to Redis. As we want to issue other Redis commands while processing a keyspace event, we'll need a second connection to Redis. Node-Redis facilitates this like so:

import { createClient } from 'redis';

const REDIS_URL = process.env.REDIS_URL || 'redis://localhost:6379';

const client = createClient({
  url: REDIS_URL
});

// First connection to Redis.
await client.connect();

// Let's duplicate that and have a second connection that we'll use for PSUBSCRIBE
const subClient = client.duplicate();
await subClient.connect();

Now we've got two connections, let's wait for keyspace pub/sub events on channels named __keyspace@0__:tokens:* (keyspace events on keys where the key name begins tokens:*):

await subClient.pSubscribe('__keyspace@0__:tokens:*', async (message, channel) => {
  console.log(`event >>> ${message} on ${channel}`);
  // Do something...
});

We're using subClient as the blocked/listening connection to Redis here... so if we want to issue further Redis commands in the callback function, we'll have to do that with client, our other connection to Redis.

We know from our keyspace notifications configuration that each event we get notified about involves some sort of change in the keyspace for keys beginning with tokens:. We can assume (because we're the ones creating the application) that all keys beginning tokens: are Redis Sets, representing the tokens that a given user has collected in our game. We can also assume that the key names are of the form tokens:<username> e.g. tokens:simon.

With those assumptions made, we can now start working on processing the notification event.

The value of channel will look something like this:

__keyspace@0__:tokens:simon

Where tokens:simon is the Redis key that was affected, and simon is the username of the player in our game. Let's parse these values out of channel:

  const affectedKey = channel.substring('__keyspace@0__:'.length);
  const userName = affectedKey.split(':')[1];

Now we have the user's username and key, we want to:

See how many tokens they currently have in their tokens set.
Update the leaderboard, creating it if necessary.
Print out the current leaderboard.

Let's look at each in turn...

We receive keyspace notification events whenever the set is modified... we know that our application's going to perform three types of operation on a user's token set:

Add a new token (adding the first token with SADD creates the set for us).
Remove a token (removing the last token with SREM also deletes the set for us).
Delete the whole set because they left the game (achieved with the DEL command).

The Redis command to get the cardinality or number of members in a set is SCARD. We can actually use this one command to cover all three of the above scenarios, without needing to know which scenario triggered the keyspace event. We can do this because SCARD returns the cardinality of the set at a given key, or 0 if the key doesn't exist. This means we don't need a separate check for the "user left the game" scenario.

With that said, let's see how many tokens are in the affected set:

const howMany = await client.sCard(affectedKey);

howMany is now going to be 0 (user left the game) or > 0 (user is still in the game and just gained or lost a token). With this information, we can now update the high score table, or create it if it doesn't alredy exist.

The high score table uses a Redis Sorted Set, which associates a value (we'll use the username here) with a score (the number of tokens) and keeps the members sorted by score.

We only need to use one of two Sorted Set commands here... if the value of howMany is > 0, we'll add the user to the high score table Sorted Set with ZADD. We're relying on a couple of useful properties of ZADD here:

If the Sorted Set doesn't already exist it will create it for us.
If the user already exists in the Sorted Set, it will update their score for us to the new value provided.

So our code for adding or updating a user in the high score table looks like this:

if (howMany > 0) {
  await client.zAdd('scoreboard', [
    { 
      score: howMany, 
      value: userName
    }
  ]);
}

And if howMany was 0, we just need to use ZREM to remove the username from the high score table (this doesn't result in an error if the high score table doesn't exist or the user isn't in it, so we don't need to check for those possibilities):

} else {
  await client.zRem('scoreboard', userName);
}

Finally, let's retrieve the entire high score table and print it out in descending score order... so that the user with the most tokens comes out on top. For this, we'll use the ZRANGE command:

console.log(await client.zRangeWithScores('scoreboard', 0, -1, {
  REV: true
}));

ZRANGE takes two zero-based index arguments... 0 being start at the first (lowest score) member in the Sorted Set, and -1 meaning return members up to and including the last (highest score). Passing in true as the value of the REV modifier tells Redis we want the results back in reverse order... so highest score down to lowest.

And that's all we need! If you're building anything with Redis, or have questions we'd love to hear from you on our Discord server.

redis-developer / keyspace-notifications-node-redis