senza update does not correctly apply changes to ASG
musiKk opened this issue · comments
I wanted to update the minimum and maximum of an ASG as well as the instance type with senza update for a dozen stacks. The instance type was updated successfully for all stacks. Unfortunately the min/max values of the ASGs have not been updated for most of the stacks. When I retry the update command for one of the stacks where it "failed" (there was never an error or something) the update is rejected with "No updates are to be performed."
Digging a bit deeper I found out that the CloudFormation templates have been updated for all stacks. So when I look at the output of senza dump or the CloudFormation designer in AWS' Management Console everything is correct. But the values have not been applied to the ASGs themselves. In my case I tried to update the min/max from 1/1 to 2/4. After the senza update for some stacks the ASGs in question still only had 1/1 whereas the CloudFormation template shows 2/4. I can see no difference between the stacks where it worked compared to those were it did not work.
I'm not sure whether this is a senza issue or not but this definitely impedes automation of certain tasks. For now I'll just fix it manually.
I recognize that there is not much tangible information in this issue but as everything went through without any error I don't know what more to say. If you can tell me ways to get more information, I'd be happy to provide it when the behavior happens again.
(stups tools are up to date as of now)
@musiKk senza update just performs a Cloud Formation update --- I don't see why the min/max ASG settings would not be propagated (unless there is/was an AWS issue). Did you check that the stacks have status "UPDATE_COMPLETE" (the CF update might take a while depending on changes)?
BTW: we are using senza update frequently to update Kubernetes AutoScaling groups, that works flawlessly: https://github.com/zalando-incubator/kubernetes-on-aws/blob/master/cluster/cluster.py#L361
Yeah, the stacks were all in UPDATE_COMPLETE state and I also waited quite some time in general. The stack did not update, I had to manually edit the ASGs in the Management Console.
I'm completely with you that this doesn't really make sense but I can only tell you what I saw. :(
Ok, I think I know now why this happened. The stacks where this failed got deployed earlier with a min/max of 2/4 but got changed via a scheduled action to 1/1. This does not change the CF template but permanently alters the ASG configuration.
@hjacobs Is there any way to change an ASG's values in this case with senza short of creating a new stack? What would be needed is something like senza scale but for min/max.
@musiKk I guess the senza scale could be improved for min/max support. But as always: you can simply use the AWS CLI to change the values directly (not as convenient).
Right, thanks!