Add support for more complex dependencies inside a pipeline
fran6co opened this issue · comments
To reduce latency we have the case of creating pipelines that have some parallel tasks inside that would make this graph:
a -> b -> d -> h -> i
-> c -> e ->
-> f -> g ->
where b, d, c,e and f,g run in parallel to each other and each one of them can be parallel or serial (so the next line waits for them accordingly). The pipeline waits for both of them to finish to move on to h.
I tried implementing this with a sub taskflow that has the parallel tasks, but it doesn't capture the dependencies the same way as it would need to make it serial if any of sub tasks are serial.
I checked Graph Processing Pipeline article but I don't see how would it work if there are more than 2 "forks" in the graph