Configure retry/consume parameters on consumer side

[Tiddo Langerak]

As far as I understand, consuming parameters can only be configured on the producer side. These are:

• consume timeout
• retry delay
• retry threshold

However, these are all concerns of the consumer, not the producer. The producer typically shouldn't know how long the consumer may take to process the message, or control the retry behaviour (consumers must be able to control this themselves to control resource utilization).

This is an even bigger issue when using fanout, as different consumers may need different parameters.

I therefore want to request a mechanism to configure these values on the consumer side, instead of the producer side.

A few possibilities:

1. Configure per consumer

The consumer already takes in a config object, we can add it there:

new Consumer({
    consumeTimeout,
    retryDelay,
    retryThreshold
});

Pro:

• Minimal changes to api surface

Con:

• No fine grained control

2. Configure per consume callback

E.g.

consumer.consume('queue', messageHandler, { consumeTimeout, retryDelay, retryThreshold }, cb);

Pro:

• More fine grained control: can be configured per queue

Con:

• Change to method signature
• Can't be controlled on a message-by-message basis

3. Return retry behaviour from message handler

We could set the retry behaviour directly from the message handler when calling it with an error:

consumer.consume('queue', (message, cb) => {
    cb(err, { retryDelay, retryThreshold });
});

// or alternative:
consumer.consume('queue', (message, cb) => {
    message.setRetryDelay();
    message.setRetryThreshold();
    cb(err);
});

Pro:

• Fine grained control on a message-by-message basis
• Each retry can change the parameters, which allows for consumers to implement advanced retry strategies like exponential back-off, or aborting completely on unrecoverable failures.

Con:

• Not a complete solution, as this doesn't work for consumeTimeout. Would therefore still need another solution for that.

4. Hook to alter message parameters

This would be called before the consume callback, and allows a consumer to set/change the parameters. E.g.:

consumer.messageParameters(msg => {
    return {
        consumeTimeout,
        retryDelay, 
        retryThreshold
    };
});

// or equivalent:

consumer.messageParameters(msg => {
    msg.setConsumeTimeout();
    msg.setRetryDelay();
    msg.setRetryThreshold();
});

Pro:

• Finegrained control
• Depending on the details, some advanced retry mechanisms might be possible.
• No (breaking) changes to existing api surface

Con:

• Potentially complicated error handling (i.e. what to do if the hook fails?)

[Tiddo Langerak]

Also more than happy to submit a PR for whichever solution we're happy with.

[weyoss]

@TiddoLangerak Thank you for opening this issue.

As far as I understand, consuming parameters can only be configured on the producer side.

I want to clarify that is not the case. The Producer does not handle or accept in anyway consuming parameters. It does not even know about their existence. Properties like (consumeTimeout, retryDelay, ttl, retryThreshold) are called "Consuming parameters" and are message properties. They can not be configured neither on the Producer side nor on the Consumer side.

These properties can be set at the Message level for a given Message instance and are handled by RedisSMQ at the Consumer side. By the way, the link you provided points out to Message API reference.

[Tiddo Langerak]

I'm sorry if I didn't explain myself properly. With "Consumer side" and "Producer side" I mean the nodes (workers/clients/applications), not the redis-smq classes.

I.e. this is the situation:

The Producer Node creates a message and calls a producer instance to send it.
The Consumer Node consumes the message using a consumer instance.

Since the Producer Node creates the message, it is also in full control over the retry/timeout parameters, and therefore controls resources of the Consumer Node.

As far as I can tell, there's no way for the Consumer Node to control retry/timeout behaviour, as the message is created by the Producer Node, and there's no mechanism for the Consumer Node to set/override these parameters. Hence, my proposals.

I hope that this clears it up?

[Tiddo Langerak]

Hi @weyoss - could you please still consider the above?