optimize dask graph computation
ungarj opened this issue · comments
Joachim Ungar commented
collect multiple tasks into larger tasks in order not to overwhelm the scheduler:
- tile batches, no overviews (
concurrent.futures
,dask
)
no dependencies - tile batches, with overviews (
dask
)
dependencies between tile batches; use tile outputs within one tile batch - tile batches with preprocessing tasks (
dask
)
dependencies between all tasks; use preprocessing output for follow up (tile) tasks