A tool for orchestrating the scale down of ECS clusters.
The downscaling algorithm is as follows:
- Find drainable container instances
- Drain
-batch-size
container instances. - Scale down ECS tasks by
-batch-size
. - Scale down ASG desired instances by
-batch-size
. - Terminate the drained container instances.
- Repeat steps 2-5 for each batch until the cluster size is at
-desired-count
. - Finally, reduce ASG maximum to the new desired count (unless
-instance-flip
is used)
GO111MODULE=on go get github.com/maikxchd/ecs-down
$ ecs-down -help
Usage of ecs-down:
-asg string
The name of the Auto Scaling Group to scale down.
-service string
The name of the ECS service to scale down.
-cluster string
The name of ECS cluster that hosts the service.
-desired-count int
The number of container instances the ECS cluster should run.
-batch-size int
The number of ECS tasks or container instances to terminate in each batch. (default 1)
-instance-type string
The container instance type that should be preferred for termination.
If not provided or if there are no instances of this type, all instances are eligible for termination.
-agent-version-before string
Prefer killing instances with agent version older than X (exclusive)
-instance-flip
Flip instances instead of scaling down EC2
-region string
The AWS region containing the resources. (default "us-west-2")
-disable-task-count
Disable task count detection
-sort-age
Sort instances in each group by instance age
Scale down box-production
to 210 instances in batches of 20. c6gd.16xlarge
container instances are selected for termination first.
ecs-down -asg prod-gql -batch-size 20 -cluster gql-production -service gql-production -desired-count 210
Scale down visage-prod
to 45 instances in batches of 5.
ecs-down -asg prod-visage -batch-size 5 -cluster visage-prod -service visage-prod -desired-count 45
Instances are selected for termination in this priority:
- If
agent-version-before
is set, these are top for termination - If
instance-type
is set, these are next priority termination - Any instances running less than some number of tasks are priority for termination (disable this group with
disable-task-count
flag) - All other instances fill the last group.
If sort-age
is used, then each sub-group is sorted so that the oldest instances are first choice. Otherwise, there is no ordering guarantee, it is whatever the API chooses to do.
In some situations it is not possible to get enough instances to say, double EC2 desired count or not plausible to get new instances rapidly and you want to repeatedley cycle out old instances in smaller quantities. For this purpose, -instance-flip
option will go towards desired ECS but keep EC2 Autoscaling Group the same size (allowing ASG to replace instances that are killed) then increases ECS count again.
As an example, in May 2020 GraphQL needed to cycle all 180 instances. To do this we set desired in ECS and EC2 to 190 and then did a bunch of runs of:
ecs-down -asg edge-platform-production-gql-production -cluster gql-production \
-service gql-production -desired-count 181 -batch-size 3 -instance-flip \
--agent-version-before "1.37.0"
Each run of the program only cycled 9 hosts (in batches of 3) allowing the instance count to always remain above 180. When the instances replaced back to 190 (because with -instance-flip
the tool never dropped the ASG desired) the program was run again until all agents were cycled.