[Question] How to avoid duplicated tasks?

[hiendaovinh]

Hi,
Does the library offer a global id to avoid duplicated tasks?

[weyoss]

Each message after being published gets a UUID. A message with an ID can not be published twice.

Additionally, let say you were in the process of publishing a message and your application crashed. Your message could be published or lost.

At the moment of the failure, if your message has been sent through the wires to Redis and has been received successfully then it will be published.

It is up to your application to manage properly such failures.

[hiendaovinh]

I'm not really talking about recovery from failures. I think having a unique job id is a legit use-case when you want to avoid duplicated tasks. E.g.
I have a prioritized queue.
I want to enqueue all tasks in an interval but sometimes I want to enqueue just some tasks. If a latter task is already in the waiting list, I expect the queue to just skip enqueuing it twice if they have the same priority.
And there's also a particular moment when I want to push a task with higher priority. I expect the queue to accept the new task, move it to the top and remove the lower-priority-and-duplicated task.

[weyoss]

Well, as I said before, the MQ uses a unique ID for each message.

As the message ID is set by the MQ you can not publish two messages with the same ID. But you can publish two messages with the same payload (body). The MQ does not care about the message payload. It is not its business.

[weyoss]

I want to enqueue all tasks in an interval but sometimes I want to enqueue just some tasks. If a latter task is already in the waiting list, I expect the queue to just skip enqueuing it twice if they have the same priority.
And there's also a particular moment when I want to push a task with higher priority. I expect the queue to accept the new task, move it to the top and remove the lower-priority-and-duplicated task.

The MQ does not work this way. Here is an example.

Considering that queue1 is a priority queue and has been already created.

import {  Producer, Message } from 'redis-smq';

const msg1 = new Message().setQueue('queue1').setBody('123').setPriority(Message.MessagePriority.HIGH);
const msg2 = new Message().setQueue('queue1').setBody('123').setPriority(Message.MessagePriority.HIGH);
const msg3 = new Message().setQueue('queue1').setBody('123').setPriority(Message.MessagePriority.LOW);

const producer = new Producer();
producer.produce(msg1, (err) => {  
 // ...
})

producer.produce(msg2, (err) => {
 // ...
})

producer.produce(msg3, (err) => {
 // ...
})

• msg1, msg2, and msg3 have the same payload.
• msg1, msg2 have the same priority
• msg3 has a different priority than msg1 and msg2

The MQ see all messages as different messages, no matter if they have the same properties or not.

All messages will be produced with different IDs and delivered with respect to their priorities. No messages will be replaced or removed.

This is the way this MQ works.

[hiendaovinh]

Thanks for the explanation but I do understand how redis-smq works. It offers at-most-once and at-last-once delivery. It's more like a suggestion than a question. Anyway, thanks for your patience!

[wack]

I believe the suggestion was: "I want to be able to manually provide an ID to a message to avoid duplicate jobs on more complex usages".