Enable dynamic modification of DataLossPrevention feature flag
juliev0 opened this issue · comments
Summary
Currently Feature flag is only read on start up. If we modify it to be dynamically modifiable, then we could modify it on all clusters and not require the Pods to be restarted.
Use Cases
When would you use this?
Message from the maintainers:
If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.
I am closing this issue altogether, as I believe the feature flag will be replaced by the new namespace level configurability. If you think it should be reopened, let me know.
reopening this since it sounds like we will use both concurrently for a little while
We can check the value each time either the PipelineRollout or the NumaflowControllerRollout or ISBServiceRollout reconcile occurs.
I've done a little thought to the scenario that something's in the middle of pausing when we turn the feature flag off. What would happen? (The reverse case of turning the feature flag on is definitely no issue)
if we suddenly turn feature flag off:
what happens to Pipeline?:
say Pipeline was paused because of us
we would notice a difference between existing pipeline (Paused) and pipeline we want (Running), then we will apply (but could take 12 minutes for a reconciliation to occur) - THIS REQUIRES THAT THE CODE not exclude lifecycle field in the comparison!
what happens to Numaflow Controller/ISBService?:
say Pipelines were paused because of us
this means that our state in memory reflects that we "want pause"
Once feature flag is off, Pipelines will notice a difference between existing pipeline (Paused) and pipeline spec (Running) and perform an update (may take 12 minutes)
What happens if the feature flag is turned back on later? There is old memory still in there representing NC/ISB's desired state which Pipelines will read. Say it says "pause" and we don't need that anymore. Pipelines will temporarily pause. At some point, the Numaflow Controller Rollout will get reconciled, however, to update that to say "pause not required", which pipelines will see and then unpause.
closing this as this feature flag has been replaced