ungarj / mapchete

Tile-based geodata processing using rasterio & Fiona

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

optimize dask graph computation

ungarj opened this issue · comments

collect multiple tasks into larger tasks in order not to overwhelm the scheduler:

  • tile batches, no overviews (concurrent.futures, dask)
    no dependencies
  • tile batches, with overviews (dask)
    dependencies between tile batches; use tile outputs within one tile batch
  • tile batches with preprocessing tasks (dask)
    dependencies between all tasks; use preprocessing output for follow up (tile) tasks