support for binary subscriptions: `pbsubscribe`, `pmessageBuffer` and utf8 issues
NickKelly1 opened this issue · comments
Hello 👋
I have an application that needs to support high pubsub throughput where the id's are cryptographic hashes so naturally binary channels and patterns are a great fit.
However it doesn't look like ioredis
supports binary patterns with psubscribe
or pmessageBuffer
, and has some issues with binary channels in general.
Problems
1. redis.on('pmessageBuffer', listener(pattern: string, channel: Buffer, payload: Buffer)
provides utf8
pattern instead of Buffer
instance
Received pmessage
messages have their pattern converted to utf8
with no option to return the original Buffer instead.
2. Internally Buffer
channels and patterns are serialized with utf8
which is not binary safe
It seems for equality (eg storing in Sets) that all binary (NodeJS Buffer
) channels and patterns are converted to utf8
string instead of a binary safe string like latin1
(ISO-8859-1).
See
Line 160 in ec42c82
For example of how this can cause issues with binary data, here are two byte sequences whose utf8
conversions from NodeJS Buffer
collide, despite the Buffer
instances containing different byte sequences.
const channel1 = Buffer.from([0xC0, 0xAF]);
const channel2 = Buffer.from([0xE0, 0x80]);
const channel1Utf8 = channel1.toString();
const channel2Utf8 = channel2.toString();
console.log(channel1Utf8); // "��"
console.log(channel2Utf8); // "��"
console.log(channel1.equals(channel2)); // false
console.log(channel1Utf8 === channel2Utf8); // true
What am I looking for?
- to use binary safe strings (eg
bufferInstance.toString('latin1')
instead ofbufferInstance.toString()
orbufferInstance.toString('utf8')
where serializing buffers to strings is necessary for equality (eg Maps and Sets) - to support passing a
Buffer
instance for thepattern
argument intopsubscribe
- to return back a
Buffer
instance instead of string for thepattern
argument oflistener
inredis.on('pmessageBuffer', listener(pattern: string, channel: Buffer, payload: Buffer)