Aborting a Message Processing when RMQ Channel Closes

[Sathish]

General Question ( So cleared off the Template )

We are using AMQPlib in Node.js to consume the messages from RMQ. In my case some times consumer takes longer than 30 mins to complete the process. But RabbitMQ server will close channel after 30 mins ( configurable at server level ), but my Node.js consumer continues to process the request which is bad, because i cannot recreate the channel now, because if recreate the same message which got timed out will be back to the consumer once again which will end up duplicate processing.

I would like to know

Is it possible to configure the channel timeout for each message or queue basis ?
Is it possible to listen for channel close and abort the already processing request in Node.js ? Because processing is not simple functions, its function of functions & even it makes database calls/redis calls internally.

[Stephen Cresswell]

Hi @sathishsoundharajan,

Is it possible to configure the channel timeout for each message or queue basis ?

Yes. You can configure the timeout per queue, either via a policy, or through an optional argument when the queue is declared, as per the RabbitMQ acknowledgement timeout documentation

Is it possible to listen for channel close

I believe so, although I'm not sure whether this will be raised as an error or close event, and if the latter, whether it will have an error argument. You'll have to investigate.

channel.on("error", (err) => {
  abort();
});
channel.on("close", (err?) => {
  abort();
});

Of course, when the channel is closed, the original message will be rolled back, so if the process takes more than 30 minutes the second time, you will be in the same situation.

Because processing is not simple functions, its function of functions & even it makes database calls/redis calls internally.

I'm sure you know what you are doing, but 30 minutes is an incredibly long time. If I remember, redis is single threaded, so commands are blocking - if the redis instance is shared this could be a problem. While I have known relational database operations to take 30 minutes or more, has always been on very large data sets (e.g. 100s of millions of rows) and when indexing had been poorly configured.

[Sathish]

@cressie176 Thank you for quick response. How can i set consumer timeout/ delivery ack timeout using amqplib ?

[Stephen Cresswell]

How can i set consumer timeout/ delivery ack timeout using amqplib ?

Call channel.assertQueue specifying the x-consumer-timeout argument as per the channel API documentation (assuming you are declaring the queue).

RabbitMQ extensions can also be supplied as options. These typically require non-standard x-* keys and values, sent in the arguments table; e.g., 'x-expires'. When supplied in options, the x- prefix for the key is removed; e.g., 'expires'. Values supplied in options will overwrite any analogous field you put in options.arguments.

[Sathish]

Hey @cressie176

One last question, is it possible to abort the on going operations when channel timeout using abort controller or something, even though channel closes the message consumed still processing. Only possible way i think of is to exit the process using process.exit so my consumer restart, create a new connection & channel consume the same message. Ideally i don't want that, instead just list for channel close and recreate the channel and reprocess the message, but i want older processing which still on-going needs to be aborted.

[Stephen Cresswell]

Hi @sathishsoundharajan,

is it possible to abort the on going operations when channel timeout using abort controller or something,

It depends what you mean by "on going operations". Assuming you mean your application functions, database calls etc, then yes, but you would have to trigger this from your close/error handler. It's nothing to do with RabbitMQ or amqplib. Using process.exit one approach, but pretty abrupt. To shutdown operations more gracefully you'll need to design it into your workflow, but it isn't easy.

instead just list for channel close and recreate the channel and reprocess the message, but i want older processing which still on-going needs to be aborted.

You can recreate the channel and start consuming messages again. But as you say, the on-going work needs to be aborted, and you will have to implement this yourself. Ideally you would want your application to pick up close to where it left off, rather from the beginning. One approach therefore is to break your workflow up into stages, and have the message consumer restart from the last successful stage.

[Sathish]

@cressie176 Thank you for patiently clarifying things. I'm kind of learning about Abortcontroller in Node.js can help solve my problem..

[Stephen Cresswell]

You're welcome @sathishsoundharajan, I hadn't actually heard of the AbortController until you mentioned it, so thank you for that. Having read some of the documentation it doesn't look very useful. I've never needed to abort a file operation, and there are many http clients I would use before turning to node-fetch, and they all have connection timeout and request timeout options. I've no idea to what extent it is being picked up by other third party libraries.

[Sathish]

Thank a lot @cressie176 .. Closing this