look into changing over to brpoplpush
tj opened this issue · comments
atomic goodness for queues, could get rid of my hack of a zpop
Any idea if/when this is coming? It looks like we could really do with this feature too!
Keep up the great work BTW and thanks! :)
I'm thinking of doing this on a major version release, specifically 1.0.0 since it requires a very big code change, and kue redesign on redis sets/lists usage.
But you currently should have no problems in many deployment situations with the existing implementation.
Thanks Behrad! afaict this will only be an issue if something fails. We need to detect that situation so we can resubmit.
when something bad fails, everything can be an issue, event with the atomic brpoplpush
As I see it, with atomic brpoplpush the work stays in redis. Without it, there's a brief windows between a pop and a push where, if the node.js process goes down, the work is lost. Am I right?
right @pgte , in the current Kue code we have seen a couple of recovery codes from this situation
A good example of how to provide priorities:
https://github.com/josiahcarlson/rpqueue/blob/master/rpqueue/__init__.py#L226
Any movement on this?
I'd planned to start the refactoring to atomic Kue 1.0 early 2015, however it was my very busy days. I'm thinking of starting it in a few months, and remember this is a really huge refactoring effecting all Kue
if you have a local redis instance (or one without poor network connection) you won't face queued stuck jobs. however active stuck jobs is something caused 95% by your app code.
Hi,
Any progress on this?
I've been very busy for a couple of months...
up ?
Hi @tj any plans to work on this?
brpoplpush is one of methods of reaching atomicity/consistency but in favor of loosing priorities...
I don't want to drop priority queues, and implementing them with brpoplpush adds some complexities.
Another method is using lua scripts which is being preferred and implemented in v1 branch currently.
@behrad. That's the point: atomicity/consistency is very important for me, priority not. Most my jobs will complete in less than 1s. Appreciate your thoughts about bee-queue?
Hello from 2018, how's 1.0.0 looking?
hello from future, our cars can flying but kue is still under 1.0.0