DXE-3327 Incorrect state after failing on roll out

[lsadlon]

Hi there,

References

#491
This ticket is to extract state issue part of above ticket

Problem Statement

When applying a policy that activates on the production network, and then subsequently changing the policy and applying that, an api error returns while the activation rolls out (expected). However, the change is accepted to staging, and then doesn't hit production. The state file is updated, so no drift is detected on subsequent runs. That then leaves us in a situation where there is no reasonable way to promote a staged policy to the production network. Not placing it in the state file wouldn't accurately reflect the fact it rolled out to staging either.

Expected Behavior

Regarding the state issue, if an error occurs, normally I would say the state should not be updated so that the drift is detected with a subsequent plan; however, what is occurring here is the staging change occurs, but the production one does not, and as a result, there is no way that I know of to re-trigger the production deployment. So, not touching state would be out of sync with staging, but touching state prevents drift for being able to trigger the production run.

Actual Behavior

Regarding the state issue, after the run fails due to an existing roll out on the production network, subsequent plans show no drift. So, we have a situation where staging is the latest version, but prod is a version behind, with no way to trigger a subsequent production deployment.

Steps to Reproduce

1. Apply a config that activates on production (policy_image or policy_video)
2. Change the policy in the terraform hcl
3. Apply that new config
4. Receive error that activation is in progress
5. Re-run apply - no drift
6. After 1 hour estimated, notice how the new policy is on staging but not on production, and there's no reasonable way to use terraform to roll it out to the production network

[Lukasz Kowalski]

This issue should be fixed with recent release.