Giters

zalando-stups / senza

Deploy immutable application stacks and create and execute AWS CloudFormation templates in a sane way

Home Page:https://pypi.python.org/pypi/stups-senza

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Senza traffic should wait for stack to be ready to change traffic

rafaelcaricio opened this issue · comments

commented

Current Behaviour

When running change traffic senza traffic on stack recently deployed. The status code of the stack is not ready ((CRETE|UPDATE)_COMPLETE) and Senza fails trying to update it.

Cause of the Problem

It is not possible to update CF stack when it's status code is not in (CRETE|UPDATE)_COMPLETE (or we should not change when it is not in one of those status code).

Expected Behaviour

Senza should wait the stack to get in a final status code if the current status of the stack is (CREATE|UPDATE)_IN_PROGRESS and ending up in some good state (CREATE|UPDATE)_COMPLETE.

commented

Is there a projected timeline, when this will be fixed?

commented

@alnkpa I am working on that right now.

commented

@alnkpa It is ready to be merged in #409

commented

Does not work as expected:

$ senza traffic kube-aws-test arp1 100
Calculating new weights.. OK
Stack Name   │Version     │Identifier                │Old Weight%│Delta│Compensation│New Weight%│Current
kube-aws-test abcdef       kube-aws-test-abcdef               0.0                            0.0         
kube-aws-test arp1         kube-aws-test-arp1                 0.0 100.0                    100.0 <       
kube-aws-test ghildebrand1 kube-aws-test-ghildebrand1         0.0                            0.0         
kube-aws-test linki19      kube-aws-test-linki19              0.0                            0.0         
Setting weights for kube-aws-test.example.org... OK
$ senza li kube-aws-test
Stack Name    │Ver.        │Status            │Created│Description                                                                                                       
kube-aws-test  abcdef       CREATE_COMPLETE    29h ago Kubernetes cluster                                                                                                 
kube-aws-test  arp1         UPDATE_IN_PROGRESS 44h ago Kubernetes cluster                                                                                                 
kube-aws-test  ghildebrand1 CREATE_COMPLETE     2h ago Kubernetes cluster                                                                                                 
kube-aws-test  linki19      UPDATE_COMPLETE     5h ago Kubernetes cluster                                                                                                 
$ senza traffic kube-aws-test linki19 50
Calculating new weights.. OK
Stack Name   │Version     │Identifier                │Old Weight%│Delta│Compensation│New Weight%│Current
kube-aws-test abcdef       kube-aws-test-abcdef               0.0                            0.0         
kube-aws-test arp1         kube-aws-test-arp1               100.0 -50.0                     50.0         
kube-aws-test ghildebrand1 kube-aws-test-ghildebrand1         0.0                            0.0         
kube-aws-test linki19      kube-aws-test-linki19              0.0  50.0                     50.0 <       
Setting weights for kube-aws-test.example.org...Validation Error: Stack:arn:aws:cloudformation:eu-central-1:123456875456:stack/kube-aws-test-arp1/6b812360-9eea-11e6-8b7e-50a68ad4f262 is in UPDATE_IN_PROGRESS state and can not be updated.
$ senza li kube-aws-test
Stack Name    │Ver.        │Status            │Created│Description                                                                                                       
kube-aws-test  abcdef       CREATE_COMPLETE    29h ago Kubernetes cluster                                                                                                 
kube-aws-test  arp1         UPDATE_IN_PROGRESS 44h ago Kubernetes cluster                                                                                                 
kube-aws-test  ghildebrand1 CREATE_COMPLETE     2h ago Kubernetes cluster                                                                                                 
kube-aws-test  linki19      UPDATE_IN_PROGRESS  5h ago Kubernetes cluster                                                                                                 
$ senza --version
Senza 2.0.163
commented

@hjacobs this logs are confusing to me. Seems like you called senza traffic kube-aws-test linki19 50 on stack link19 and the stack weight changed for the stack called arp1.

Anyway, seems like we are calling to change the stack twice in the set new weights code. I will take a deeper look on that.

commented

Oh, I see what is the problem. Will fix that! All the versions of the deployed stack we want to change the weights should be in a final state, not only the version we mention in the CLI.

commented

Actually all that have weights >0.0.

commented

Any updates?

commented

@rafaelcaricio any way to support you on that?

commented

I started working on this but haven't had time to finish. Probably next week since now things are calming down. Sorry about that.