redis / ioredis

🚀 A robust, performance-focused, and full-featured Redis client for Node.js.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

support for binary subscriptions: `pbsubscribe`, `pmessageBuffer` and utf8 issues

NickKelly1 opened this issue · comments

Hello 👋

I have an application that needs to support high pubsub throughput where the id's are cryptographic hashes so naturally binary channels and patterns are a great fit.

However it doesn't look like ioredis supports binary patterns with psubscribe or pmessageBuffer, and has some issues with binary channels in general.

Problems

1. redis.on('pmessageBuffer', listener(pattern: string, channel: Buffer, payload: Buffer) provides utf8 pattern instead of Buffer instance

Received pmessage messages have their pattern converted to utf8 with no option to return the original Buffer instead.

See https://github.com/redis/ioredis/blob/ec42c82ceab1957db00c5175dfe37348f1856a93/lib/DataHandler.ts#L143C1

2. Internally Buffer channels and patterns are serialized with utf8 which is not binary safe

It seems for equality (eg storing in Sets) that all binary (NodeJS Buffer) channels and patterns are converted to utf8 string instead of a binary safe string like latin1 (ISO-8859-1).

See

const channel = reply[1].toString();

For example of how this can cause issues with binary data, here are two byte sequences whose utf8 conversions from NodeJS Buffer collide, despite the Buffer instances containing different byte sequences.

const channel1 = Buffer.from([0xC0, 0xAF]);
const channel2 = Buffer.from([0xE0, 0x80]);

const channel1Utf8 = channel1.toString();
const channel2Utf8 = channel2.toString();

console.log(channel1Utf8); // "��"
console.log(channel2Utf8); // "��"

console.log(channel1.equals(channel2)); // false
console.log(channel1Utf8 === channel2Utf8); // true

What am I looking for?

  1. to use binary safe strings (eg bufferInstance.toString('latin1') instead of bufferInstance.toString() or bufferInstance.toString('utf8') where serializing buffers to strings is necessary for equality (eg Maps and Sets)
  2. to support passing a Buffer instance for the pattern argument into psubscribe
  3. to return back a Buffer instance instead of string for the pattern argument of listener in redis.on('pmessageBuffer', listener(pattern: string, channel: Buffer, payload: Buffer)