jayhack / prefect

The easiest way to coordinate your dataflow

Home Page:https://prefect.io

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Prefect task fails if dask worker is killed even if restarts is enabled

jayhack opened this issue · comments

First check

  • I added a descriptive title to this issue.
  • I used the GitHub search to find a similar issue and didn't find it.
  • I searched the Prefect documentation for this issue.
  • I checked that this issue is related to Prefect and not one of its dependencies.

Bug summary

Killing a dask worker running prefect task in Kubernetes cluster results in the error below, rather than task getting restarted

Reproduction

Any minimal flow with dask_kubernetes and prefect_dask

Error

17:38:18.808 | INFO    | Task run 'test-636' - Crash detected! Execution was interrupted by an unexpected exception: concurrent.futures._base.CancelledError: test-636-f0e8cc3c35094b0785479d81317b3f2b-1   │
│ 17:38:20.836 | INFO    | Task run 'test-637' - Crash detected! Execution was interrupted by an unexpected exception: concurrent.futures._base.CancelledError: test-637-8c49c5455ba84b1282405b67cb7bea08-1   │
│ 17:38:21.992 | INFO    | Task run 'test-638' - Crash detected! Execution was interrupted by an unexpected exception: concurrent.futures._base.CancelledError: test-638-5073a6460b2b42a382f49e945725f904-1   │
│ 17:38:23.186 | INFO    | Task run 'test-639' - Crash detected! Execution was interrupted by an unexpected exception: concurrent.futures._base.CancelledError: test-639-d836a5af54984a9c843d2e87f068f585-1   │
│ 17:38:24.237 | INFO    | Task run 'test-640' - Crash detected! Execution was interrupted by an unexpected exception: concurrent.futures._base.CancelledError: test-640-fbd596b3ec384b99ada78c355f7ccc0e-1

Versions

Version:             2.8.5
API version:         0.8.4
Python version:      3.11.2
Git commit:          81a67202
Built:               Thu, Mar 9, 2023 4:27 PM
OS/Arch:             linux/x86_64
Profile:             default
Server type:         server

Additional context

No response