Parallel execution based on dependency tree
tempora-mutantur opened this issue · comments
Hello.
We do have long running tasks that we want to run faster, like everyone. We benefit drastically from parallel tasks execution and would like the project to be able to build some sort of a DAG so that the tool could automatically understand what tasks/dependencies it can run in parallel.
Similar features exist in the following projects:
https://github.com/magefile/mage/blob/master/site/content/dependencies/_index.en.md#parallelism
https://taskfile.dev/#/usage?id=task-dependencies
https://pydoit.org/cmd-run.html#parallel-execution
I think, it's a big feature and many users could benefit from it.
Very thank you.
I started thinking about this feature from the year one of Invoke-Build, especially on development of Build-Parallel.
Because it looks somewhat natural, academically interesting, and "can do this" (probably, with some caveats).
Now it's the second decade of Invoke-Build and the feature is still not there :)
And full disclose, I am not planning it until I see frequent and solid use cases.
Reasons (excuses?)
(1) Many tasks are fast, running them all in parallel without thinking may be slower due to inevitable overhead of parallelism.
(2) Many tasks involve file reading and writing, parallel execution of them is not necessarily faster.
(3) Many tasks are formally independent by design and yet cannot be parallel.
Example: a script with by design independent tasks build, test.
They are independent because each of them may be invoked on its own.
Maybe not always, e.g. test cannot be run when nothing is built.
But when it's built one can and often runs just test (working on tests).
(4) Composing scripts knowing that every task is potentially parallel with who knows what (now or later when tasks are added or changed) is so much harder.
It is not possible to have a natural task build-and-test composed as build, test from example (3).
Not to mention constant thinking of shared state, variables, files, resources, etc.
What to do
Understand what tasks really consume build time and can be parallel, then make them parallel manually, e.g. by Build-Parallel.
Boring? Probably. But in reality this approach may work much better, save time including development, with less problems.
Note that problems in parallel scenarios are not always discoverable right away.
Use cases?
Like I said, if there are frequent and solid use cases... who knows... Do you have some "not too contrived", but ideally real, build script examples that might benefit from this feature?
Thank you for the response, I see the reasons you decided to avoid implementing it. At the moment we're using a similar approach with one you suggested with Build-Parallel. And what we found that there are many places which are not executed efficiently and the best approach to handle it is building this dependencies graph. What we see is making dependencies run in parallel would make our procedures faster 1.5 times. But I see how this increases tool's complexity significantly and if there's no that much demand this may not worth it. Thank you.