Giters

LeanerCloud / AutoSpotting

Saves up to 90% of AWS EC2 costs by automating the use of spot instances on existing AutoScaling groups. Installs in minutes using CloudFormation or Terraform. Convenient to deploy at scale using StackSets. Uses tagging to avoid launch configuration changes. Automated spot termination handling. Reliable fallback to on-demand instances.

Home Page:https://autospotting.io

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

autospotting not respecting presence of ephemeral volume in AMI/launch template

karock opened this issue · comments

commented

Github issue

Issue type

  • Bug Report

Build number

nightly-1137

believe we used master branch code from end of April 2020.

Environment

  • AWS region: us-east-1
  • Type of environment: VPC

Summary

I've got an ASG set up to launch instances from a Launch Template using m5d.xlarge instances. However autospotting is allowing spot instances without an ephemeral volume to be used.

I've attempted to detail my setup below, however if there's some config I've missed that makes the ephemeral volume requirement apparent to autospotting in a different way than I attempted please do let me know! :)

Steps to reproduce

The AMI used for this template requests an ephemeral volume:

Block Devices
/dev/xvda=snap-0a74061116e945b70:16:true:gp2, /dev/sdb=ephemeral0

The launch template says: Currently no volumes details are specified and therefore the instances launch with the template default volume settings. When I go to modify the template I can only add EBS volumes, but the instance store volume is listed:

Screen Shot 2020-05-13 at 11 40 40 AM

Debug output from autospotting shows:

2020/05/13 12:54:43 instance.go:373: Comparing current type m5d.xlarge with price 0.226 with candidate m5.xlarge with price 0.0815
2020/05/13 12:54:43 instance.go:327: Checking allowed/disallowed list
2020/05/13 12:54:43 instance.go:219: Comparing class spot/instance:
2020/05/13 12:54:43 instance.go:220: 	Spot CPU/memory/GPU:  4  /  16  /  0
2020/05/13 12:54:43 instance.go:222: 	Instance CPU/memory/GPU:  4  /  16  /  0
2020/05/13 12:54:43 instance.go:286: Comparing storage spot/instance:
2020/05/13 12:54:43 instance.go:287: 	Spot volumes/size/ssd:  0 0 false <-----------
2020/05/13 12:54:43 instance.go:291: 	Instance volumes/size/ssd:  0 150 true <-----------
2020/05/13 12:54:43 instance.go:312: Comparing virtualization spot/instance:
2020/05/13 12:54:43 instance.go:313: 	Spot virtualization:  [HVM]
2020/05/13 12:54:43 instance.go:314: 	Instance virtualization:  hvm
2020/05/13 12:54:43 instance.go:383: 	MATCH FOUND, added m5.xlarge to launch candiates list

I took a quick look through the code and came upon:

	usedMappings := i.asg.launchConfiguration.countLaunchConfigEphemeralVolumes()
	attachedVolumesNumber := min(usedMappings, current.instanceStoreDeviceCount)
...
	i.isStorageCompatible(candidate, attachedVolumesNumber) &&

Stands to reason one of those terms is 0 and the min is going with that. I suspect it's usedMappings but not entirely sure.

Expected results

Expecting launch template with AMI having ephemeral volume specified to require instance types that have an ephemeral volume available.

Actual results

commented

Good catch!

Looks like the Launch Template block device mappings are not handled, we just do it for groups that use the Launch Configuration. Likely the number returned for Launch configuration is zero if you are using Launch Templates.

We'll need to also count the block device mappings of the Launch Template and use it for determining the used block device mappings.

commented

Cool, suspected that might've been the case. Let me know if I can help in any way. Not really proficient with Go, unfortunately.

commented

Unfortunately I don't have much time these days, but it should not be so hard, just give it a try.