Priority job queue
ahmadferdous opened this issue · comments
Having read the documentation, I got the impression that the job queue that the workers receive job from is FIFO. In many applications, the jobs have priority associated. If I want to make job queue type configurable (FIFO, priority queue), how much design change in verk do you think one would have to make?
I would simply have multiple queues for different needs. It's a lot easier to have queues for each different priority (low, high, etc) for your application. I wouldn't build such feature as part of Verk TBH.
Thanks for your response.
With verk, is it possible to have one worker (or a pool or workers doing the same thing) that all consume from the same set of job queues? For example, let's assume we have 3 queues for 3 different priorities (low, medium, high) for the same kind of job. We have 2 instances of the same worker, who will get jobs according to priority. So, jobs in high priority will be executed first. Jobs in medium queue will be processed only if there's nothing in high priority queue. This kind of scenario is relevant when we have rate limiting constraint and we want to serve high priority clients first.
I could not figure out the answer from README file and that's why I'm asking here.
No there's no such thing but you can achieve something similar by having more workers for the high priority. For example:
- low - 1 worker
- medium - 5 workers
- high - 10 workers
So up to 10 concurrent jobs will run for the high queue, 5 for the medium and just 1 for low.
Sadly it won't use all workers if just low jobs are available.