Nested batches
arjenbrandenburgh opened this issue · comments
I noticed in the "Projects" that this is a wanted feature.
Any chance of this being done soon? If not, could you point me in the right direction on how you would see this being developed? What the right approach should be?
The problem I'm trying to solve is as follows:
WorkflowWorker will spawn 1 overall batch, which will spawn X batches (from database, so don't know how many).
Each batch will create a new batch of 1 worker, which when finished (callback), will start a new batch with X workers (from database, so don't know how many).
When everything's finished, the WorkflowWorker overall batch should call a callback, spawning a final worker.
So, basically. If this implementation succeeds, I would be able to do what I want.
Update: taking a crack at this. Really close...
So @cybey I didn't think it over much before but maybe the simplest solution is to pass the ids of the parent batches to a child one and just add the job id to the parent batch (including child batch too). Also you'll have take care of notifying the parent batch to know when the jobs are ready.
Let me know if you need some help.
https://github.com/cybey/sidekiq-batch/commits/nested_batches
Code is still a bit rough. It seems to be working for completed and successful events. However, failures still don't work as they should. At least, the callback "complete" isn't triggered
Also need to rewrite a lot of tests...
Would be appreciated if you can have a look at what I did.
Tests have been modified and a few added. The issue I had with failed callbacks has been resolved.
Will go through it again tomorrow and create a PR for it if you don't mind.
That's great! Sure, open a PR whenever you think it's ready. We can discuss it then in the PR.
Have opened a PR at #6
Thanks @cybey will look into it.
Closing as 0.1.3 was released. Appreciate @cybey