Deletion of job from queue does not clear the lock key
driv3r opened this issue · comments
When you try to remove the unique job from within Web UI, the lock key is not cleared from redis. Also the same thing can occur when deleting the job or clearing the whole queue from ruby. Probably one of the solutions would be to extend the sidekiq api to also clear the lock first, or at least gather the locks for unlock. If I will have a spare second I can take a deeper look into it.
Hello, yes, you are right and it should be fixed. Unfortunately i can't take a look into this now, so any help would be appreciated. Thanks!
The main issue that I see is that the two things are different. In one case you have this key that represents the uniqueness of a job and on the other you have the record of the job. When everything in working normally you go through the right channels and it works as expected but through the command line you can only do so much to keep the lock status updated.
The web ui part should be fixed but otherwise i feel like we are all out of luck with that.
Yes, normally the stuff goes through the middleware, but there's a API so you can fetch a whole job, browse through them and delete. The thing is that with the current implementation, the only thing that can be a bit slow, is deletion of all the jobs. It's because normally it will just clear the redis without checking each job, and we will need to change that into deletion of the jobs one after another.
I already was looking into the tests and implementation but haven't got enough time till now. Probably on this weekend I will try to close it.
Besides that I was wondering why the unique key generation is implemented as separate standalone module instead of something included within the worker class?
I wrote up a quick fix for this, but it needs the webserver to require all the relevant Job files one way or the other, and there's no way (that I know of) to ensure that via a standalone gem like this. Any thoughts?
Hey @dychen, why do you think this is not possible? If sidekiq-middleware is required after sidekiq it can override some of its code. So, basically, sidekiq-middleware should be required in a sidekiq web interface environment in order to hijack job deletion. Other way (and preferred for me) would be implementing this feature in a separate file (which is not required by default) and ask users to require it manually in web interface envrironment.
What do you think?
Yeah, that's actually the way I'm doing it (the first way you mentioned), and it works. The caveat is that now you have to make sure that the jobs are required somewhere on the webserver load path (maybe config.ru), whereas before, the webserver didn't need to know about the jobs at all. This is because before, it was just deleting the jobs from Redis - now, it's actually calling unlock! on the jobs because it needs to find the job lock and that logic is handled by the worker. This is a little concerning (at least, in my opinion) because there's no way to make sure the gem user does this. I can submit a PR if you wanna take a look.
@dychen, I think that your solution would work great. Since we prefer implementing it as a separate extension that should be manually required, this just needs to be well documented.
using sidekiq-pro: if I stop the sidekiq process while an unique task is running, when I restart sidekiq the job is performed twice. Is this a similar issue?