zalando-stups / senza

Deploy immutable application stacks and create and execute AWS CloudFormation templates in a sane way

Home Page:https://pypi.python.org/pypi/stups-senza

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Traffic switch affected by older stacks being deleted in background

a1exsh opened this issue · comments

Just had a new version of the test stack deployed via CI/CD. In the CD deploy step the traffic is switched automatically 100% to the newest version and the oldest one is scheduled for deletion.

Noticed a problem in the newest version, so wanted to switch traffic back to old stable one. Observed the current traffic report:

$ senza traffic abcdef-test
Stack Name │Version│Identifier       │Weight%
abcdef-test cd209   abcdef-test-cd209     0.0 
abcdef-test cd210   abcdef-test-cd210     0.0 
abcdef-test cd212   abcdef-test-cd212   100.0 

Issued command to move 100% of the traffic from cd212 back to cd210. The oldest one, cd209 should not be involved in any way, however:

$ senza traffic abcdef-test cd210 100
Calculating new weights.. OK
Stack Name │Version│Identifier       │Old Weight%│Delta │Compensation│New Weight%│Current
abcdef-test cd209   abcdef-test-cd209         0.0                             0.0         
abcdef-test cd210   abcdef-test-cd210         0.0  100.0                    100.0 <       
abcdef-test cd212   abcdef-test-cd212       100.0 -100.0                      0.0         
Setting weights for abcdef-test.acid.zalan.do...Validation Error: Stack:arn:aws:cloudformation:eu-central-1:1234567890:stack/abcdef-test-cd209/8ca341d0-0278-11e7-9750-500c52a6ce9a is in DELETE_IN_PROGRESS state and can not be updated.

The update of cd210 and cd212 still happened, but the error here seems to be avoidable. We know that we need to move the traffic only between the two stacks, cd209 should not be touched at all.