Parsl / parsl

Parsl - a Python parallel scripting library

Home Page:http://parsl-project.org

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

some kind of stack depth RecursionError on fast running tasks/ join apps

benclifford opened this issue · comments

Describe the bug

This is a placeholder for a RecursionError I've seen occasionally in the recursive fibonacci pytest; and (I think the same) that @svandenhaute has seen in a real application.

This seems to be coming from something like:

When a task is submitted, later on (a few lines of code later) a callback is added to handle that task completing. If the task has already completed by the time that callback is added, the next task launch code will run inside that add_done_callback call, with the stack still having everything else above it; and this can happen to arbitrary depth, I think (given arbitrarily deep task dependencies?)

I think this is roughly to do with fast running tasks and some large number of callbacks (eg depdendency depth?)

This issue a place holder to collect more info on this bug - especially stack traces and logs - to help understand it.

To Reproduce
not yet