Prefect task fails if dask worker is killed even if restarts is enabled
jayhack opened this issue · comments
First check
- I added a descriptive title to this issue.
- I used the GitHub search to find a similar issue and didn't find it.
- I searched the Prefect documentation for this issue.
- I checked that this issue is related to Prefect and not one of its dependencies.
Bug summary
Killing a dask worker running prefect task in Kubernetes cluster results in the error below, rather than task getting restarted
Reproduction
Any minimal flow with dask_kubernetes and prefect_dask
Error
│ 17:38:18.808 | INFO | Task run 'test-636' - Crash detected! Execution was interrupted by an unexpected exception: concurrent.futures._base.CancelledError: test-636-f0e8cc3c35094b0785479d81317b3f2b-1 │
│ 17:38:20.836 | INFO | Task run 'test-637' - Crash detected! Execution was interrupted by an unexpected exception: concurrent.futures._base.CancelledError: test-637-8c49c5455ba84b1282405b67cb7bea08-1 │
│ 17:38:21.992 | INFO | Task run 'test-638' - Crash detected! Execution was interrupted by an unexpected exception: concurrent.futures._base.CancelledError: test-638-5073a6460b2b42a382f49e945725f904-1 │
│ 17:38:23.186 | INFO | Task run 'test-639' - Crash detected! Execution was interrupted by an unexpected exception: concurrent.futures._base.CancelledError: test-639-d836a5af54984a9c843d2e87f068f585-1 │
│ 17:38:24.237 | INFO | Task run 'test-640' - Crash detected! Execution was interrupted by an unexpected exception: concurrent.futures._base.CancelledError: test-640-fbd596b3ec384b99ada78c355f7ccc0e-1 │
Versions
Version: 2.8.5
API version: 0.8.4
Python version: 3.11.2
Git commit: 81a67202
Built: Thu, Mar 9, 2023 4:27 PM
OS/Arch: linux/x86_64
Profile: default
Server type: server
Additional context
No response