jayhack / prefect

The easiest way to coordinate your dataflow

Home Page:https://prefect.io

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Allow flow run idempotency keys to be updated

jayhack opened this issue · comments

First check

  • I added a descriptive title to this issue.
  • I used the GitHub search to find a similar request and didn't find it.
  • I searched the Prefect documentation for this feature.

Prefect Version

2.x

Describe the current behavior

Currently there is patch request (/api/flow_runs/{id}) for flow run that exposes possibility to change some fields but not idempotency key.

Describe the proposed behavior

Expose the idempotency key in the patch /api/flow_runs/{id}

Example Use

Idempotency key is used in case we do not want to have multiple flow runs for a given flow + some internal logic. In some cases one might want to change the idempotency key for the old run to have the possibility to do a fresh rerun.

Additional context

Currently if we use idempotency key we can reset the state of flow_run with api/flow_runs/{id}/set_state this has following side effects:

  • logs of new flow run is linked to the previous one
  • tags are shared
  • parameters like start time are common

Sometimes the above is desirable, but in some cases it might be more expected to have separation between the two runs. For that we either have to give new unique idempotency key to the new flow run or change the one for the old one.

  • First point is not an option as we want to have possibility to call POST api/flow_runs with idempotency_key that is precalculated without querying prefect. We want to just call POST api/flow_runs with idempotency_key and have new flow_run registered if there was none / old was "calcelled/renamed" or have current one returned if is there.
  • Remove / change idempotency key from old one so that POST api/flow_runs with idempotency_key would create new flow_run while called for the first time and the next calls would not create new flow_runs but just return existing one.

Related issues:
PrefectHQ#8321