Allow skipping a workflow if mutex cannot be acquired
nettrino opened this issue · comments
Summary
If a mutex is at the workflow level, other workflows with the same mutex will wait till the mutex is acquired. However, there would be cases where we would want to skip a workflow entirely if another instance of the workflow has the mutex. This could be doable via some configuration option like the following:
synchronization:
mutex:
name: test
acquireFailurePolicy: skipWorkflowWe could expose different types of policies like skipWorkflow, wait, etc.
Use Cases
There are cases were a workflow runs to ingest / update some data set, for instance fetch new rows from a database for a given entity. Running two of the same workflows will have no meaning as they will fetch the exact same data, thus a mutex prevents this duplication, however currently it does not prevent duplicating work back-to-back without custom logic within the workflow executable. A policy that would determine what to do if a lock is not acquired would allow skipping the secondary (redundant) workflow entirely.
Beyond this issue, one could allow more complex blocking policies as well, where a mutex is considered to be "held" for longer periods of time than a workflow is running for. For instance, say I ingest data that get updated every 12h from some remote source. If a single one of the workflows has been run within that time window, we could force the mutex to be held for 12h, even if the workflow has been completed, so as no other workflows can run and do redundant work within that time period. This would be useful because, even if skipWorkflow is accepted as a feature request, one could schedule redundant workflows to be running one after the other without lock contention.
Message from the maintainers:
Love this feature request? Give it a 👍. We prioritise the proposals with the most 👍.
