Analysis organizational principles in practice: dask / luigi and the line in the sand between the two
lgray opened this issue · comments
Dask and Luigi (and their contemporaries) are both extremely flexible tools that provide ways to generate DAGs. Dask is often well situated to describe the data-flow and mathematical operations on some inputs, and Luigi is engineered more to isolate and interlink larger conceptual chunks of "stuff to do" into a complete workflow. Either can achieve the functionality of the other, but it's clear there's some boundary using both which achieves a clean and focusing division of responsibilities.
Let's talk about it. :-)
+1
+1
+1
+1
+1
+1
That's interesting! Maybe we can also bring snakemake into the game? It's used a lot in LHCb at least
+1
+11
+1