processing pauses on first empty queue
kcburge opened this issue · comments
I don't have a fix for this, but, the poll logic will pause on the first empty queue, so, if you're processing more than one queue, one with data and one empty, it will pause 5 seconds after processing each job in the queue. Easy enough to work around by specifying exactly one queue, but, definitely caught me by surprise.
@kcburge the pausing is at the worker level. Create a worker per queue to avoid intermittent poll pausing. Make sense?
In the readme, it has the example:
// setup a worker
var worker = require('coffee-resque').connect({
host: redisHost,
port: redisPort
}).worker('*', myJobs)
So, I created a worker to wait on "asterisk", which causes the polling as it alternated between the queues, as I described. I would change the example and recommend not waiting on "asterisk".
I'll dig into this a little deeper over the weekend when I have some free time.
On Jun 15, 2012, at 9:46 AM, Kevin Burge wrote:
In the readme, it has the example:
// setup a worker
var worker = require('coffee-resque').connect({
host: redisHost,
port: redisPort
}).worker('*', myJobs)So, what I created a worker to wait on "", which causes the polling as it alternated between the queues, as I described. I would change the example and recommend not waiting on "".
Reply to this email directly or view it on GitHub:
#22 (comment)
Sean McDaniel
sean.mcdaniel@mac.com
@kcburge Finally got around to this. Using a splat '*' as the queue arg has a few drawbacks with the current implementation:
- The reason that you have outlined here wrt the polling behavior. If you don't have a high volume of jobs then maybe this is ok and you can simply decrease the optional timeout config param.
- It really isn't dynamic. Determining the set of queues for the worker is performed once during instantiation. Enqueing new jobs on a new queue will not get picked up dynamically by a existing worker. In addition, and this is a bigger concern, if the queues don't exist in redis when the worker is initialized well then the worker is actual not bound to any queue.
In the systems that I have worked on that use coffee-resque I use a single queue that for several different job types. This keeps my workers busy while there are jobs available, ie they don't incur the pause penalty.
In any case I have a couple of ideas to resolve these issues that I'm going to try out soon.
No rush for my sake. It has been working fine for me without the splat.
