Senza respawn instances doesn't prevent AutoScaling from trying to compensate for terminated nodes
a1exsh opened this issue · comments
When trying senza respawn command (after updating the CloudFormation stack definition using senza update) I've seen that there was an unexpected instance created in the AutoScaling group.
The initial min/max capacity of the ASG was 1 and from the output of senza I would expect that AutoScaling should not react to instance termination before the respawn procedure is complete for all nodes.
The senza definition was using Spot instances. In the Spot Requests history I've seen a 'cancelled' request for the node. In the AutoScaling group Activity History tab there was a scaling event recored with 'Cancelled' status as well. The 'Cause' of event was:
At 2016-11-18T09:43:08Z a difference between desired and actual capacity changing the desired capacity, increasing the capacity from 1 to 2.
The extra instance was running for some more minutes, then it was gone. The respawn command completed w/o reporting any errors and otherwise everything was fine.
Maybe instead of suppressing scaling policies and terminating old instances manually one could use OldestInstance termination policy: http://docs.aws.amazon.com/autoscaling/latest/userguide/as-instance-termination.html#custom-termination-policy