invalid memory address or nil pointer dereference: errorString lambdaPanicResponse

Question

invalid memory address or nil pointer dereference: errorString lambdaPanicResponse

kulinsj opened this issue 3 years ago · comments

Jan commented 3 years ago

Github issue

Issue type

Bug Report

Build number

nightly-24b40a8

Configuration

Default stack config

Environment

Stack in us-east-1
spot instances attempting spot request replacement are in us-west-2
VPC
Anonymized launch configuration:

instance_type = "m3.medium"
root_block_device {
  volume_size = "33"
}
lifecycle {
  create_before_destroy = true
  ignore_changes        = [user_data]
}

Summary

Installed the evaluation stack with the default settings. The regional lambdas trigger and send the message to SQS, SQS forwards the data to the other Lambda. During its execution, the second Lambda function runs into runtime error: invalid memory address or nil pointer dereference: errorString

Steps to reproduce

Install evaluation stack with default settings

Expected results

Replace existing on-demand instances with spot requested instances


START RequestId: c0a8f0f7-18e6-57b7-b62d-a9cf26466fdb Version: $LATEST
2021/06/07 15:34:05 autospotting.go:46: Starting autospotting agent, build nightly-24b40a8 expiring on 28-Jun-2021
2021/06/07 15:34:05 autospotting.go:53: Configuration flags: autospotting.Config{AutoScalingConfig:autospotting.AutoScalingConfig{MinOnDemandNumber:0, MinOnDemandPercentage:0, AllowedInstanceTypes:"*", DisallowedInstanceTypes:"", OnDemandPriceMultiplier:1, SpotPriceBufferPercentage:10, SpotProductDescription:"Linux/UNIX (Amazon VPC)", SpotProductPremium:0, BiddingPolicy:"normal", TerminationMethod:"", InstanceTerminationMethod:"autoscaling", TerminationNotificationAction:"auto", CronSchedule:"* *", CronTimezone:"UTC", CronScheduleState:"on", PatchBeanstalkUserdata:"", GP2ConversionThreshold:170}, InstanceData:(*ec2instancesinfo.InstanceData)(0xc000399968), LogFile:(*os.File)(0xc0000aa008), LogFlag:19, Regions:"ap-northeast-1,ap-northeast-2,ap-south-1,ap-southeast-1,ap-southeast-2,ca-central-1,eu-central-1,eu-north-1,eu-west-1,eu-west-2,eu-west-3,sa-east-1,us-east-1,us-east-2,us-west-1,us-west-2", MainRegion:"us-east-1", SleepMultiplier:1, FilterByTags:"", TagFilteringMode:"opt-in", Version:"nightly-24b40a8", LicenseType:"evaluation", PatchBeanstalkUserdata:"false", EventFile:"", FinalRecap:map[string][]string{}, SQSQueueURL:"https://sqs.us-east-1.amazonaws.com/255827576516/AutoSpotting.fifo", sqsReceiptHandle:""}
2021/06/07 15:34:05 main.go:185: Received event:
{"Records":[{"messageId":"dfff1935-6a8a-442c-b868-4aa5da2d0a61","receiptHandle":"AQEBj8VqUF90815Wy8sz6qA7bb7OLjNoWt1d+PT9u63C0JebSZo0qhfJil5xWJzyZ1CqBFnSQ8CT5t2UPQNOxw44EJgGxsOofF7aWgdlEc+MlclXcnpF+O9rrMMWZMhji2qsyMb7/E8IjTSLdI3kUrgymBaHkmtvq8ctkm3hTvUqRzcuxE4+JYq9WmrvIa3zjizf4Clr25ZQ9JoYZr0m2HA0qAbY1mADuTU7bGw7EOFRIVQC5XOAbvHMJEbMt6dWpvOedJbYWDG1QfarFS6n7eMT8A==","body":"{\"version\":\"0\",\"id\":\"890abcde-f123-4567-890a-bcdef1234567\",\"detail-type\":\"EC2 Instance State-change Notification\",\"source\":\"aws.events\",\"account\":\"\",\"time\":\"2021-06-07T15:34:01Z\",\"region\":\"us-west-2\",\"resources\":[\"arn:aws:events:us-east-1:123456789012:rule/SampleRule\"],\"detail\":{\"instance-id\":\"i-00cce956596d3173b\",\"state\":\"running\"}}","attributes":{"ApproximateReceiveCount":"1","SentTimestamp":"1623080041123","SequenceNumber":"18862252564237039872","MessageGroupId":"us-west-2-staging-asg","SenderId":"AROATXEEDA3CLII2HCUSY:AutoSpotting-LambdaFunction-NK30c2SDHVgB","MessageDeduplicationId":"153cc0e1d4bf1641769431a5993cf8f553b487f41bb0a0c46a8e8649e1d9c3d5","ApproximateFirstReceiveTimestamp":"1623080041123"},"messageAttributes":{},"md5OfBody":"2a8203064a6431dd720603969ef47bbc","eventSource":"aws:sqs","eventSourceARN":"arn:aws:sqs:us-east-1:255827576516:AutoSpotting.fifo","awsRegion":"us-east-1"}]}
2021/06/07 15:34:05 main.go:253: Triggered by EC2 Instance State-change Notification
2021/06/07 15:34:06 region.go:417: Enabling group staging-asg for processing because its tags, the currently configured filtering mode (opt-in) and tag filters are aligned
2021/06/07 15:34:06 region.go:417: Enabling group thumbor-asg for processing because its tags, the currently configured filtering mode (opt-in) and tag filters are aligned
2021/06/07 15:34:06 main.go:377: Scanning full instance information in us-west-2
2021/06/07 15:34:06 spot_price.go:25: us-west-2 Requesting spot prices
2021/06/07 15:34:07 main.go:392: us-west-2 Found instance i-00cce956596d3173b in state running
2021/06/07 15:34:07 autoscaling.go:329: Adding instances to staging-asg
2021/06/07 15:34:07 autoscaling_configuration.go:447: No default value for on-demand instances specified, skipping.
2021/06/07 15:34:07 autoscaling_configuration.go:263: Couldn't load the GP2ConversionThreshold from tag autospotting_gp2_conversion_threshold, using the default value of 170
2021/06/07 15:34:07 instance.go:266: us-west-2 instace i-00cce956596d3173b belongs to enabled ASG staging-asg
2021/06/07 15:34:07 autoscaling.go:752: staging-asg Counting already running on-demand instances
2021/06/07 15:34:07 autoscaling.go:765: staging-asg Found 2 on-demand instances running on a total of 2
2021/06/07 15:34:07 autoscaling.go:133: Currently more than enough OnDemand instances running
2021/06/07 15:34:07 main.go:434: us-west-2 instance i-00cce956596d3173b belongs to an enabled ASG and should be replaced with spot, attempting to launch spot replacement
2021/06/07 15:34:07 launch_configuration.go:32: Launch configuration would attach 0 ephemeral volumes if available
2021/06/07 15:34:07 instance.go:495: MATCH FOUND, added c3.large to launch candidates list for instance i-00cce956596d3173b
2021/06/07 15:34:07 instance.go:495: MATCH FOUND, added c3.xlarge to launch candidates list for instance i-00cce956596d3173b
2021/06/07 15:34:07 instance.go:495: MATCH FOUND, added c4.large to launch candidates list for instance i-00cce956596d3173b
2021/06/07 15:34:07 instance.go:495: MATCH FOUND, added c4.xlarge to launch candidates list for instance i-00cce956596d3173b
2021/06/07 15:34:07 instance.go:495: MATCH FOUND, added c5.large to launch candidates list for instance i-00cce956596d3173b
2021/06/07 15:34:07 instance.go:495: MATCH FOUND, added c5.xlarge to launch candidates list for instance i-00cce956596d3173b
2021/06/07 15:34:07 instance.go:495: MATCH FOUND, added c5a.large to launch candidates list for instance i-00cce956596d3173b
2021/06/07 15:34:07 instance.go:495: MATCH FOUND, added c5a.xlarge to launch candidates list for instance i-00cce956596d3173b
2021/06/07 15:34:07 instance.go:495: MATCH FOUND, added c5ad.large to launch candidates list for instance i-00cce956596d3173b
2021/06/07 15:34:07 instance.go:495: MATCH FOUND, added c5ad.xlarge to launch candidates list for instance i-00cce956596d3173b
2021/06/07 15:34:07 instance.go:495: MATCH FOUND, added c5d.large to launch candidates list for instance i-00cce956596d3173b
2021/06/07 15:34:07 instance.go:495: MATCH FOUND, added c5d.xlarge to launch candidates list for instance i-00cce956596d3173b
2021/06/07 15:34:07 instance.go:495: MATCH FOUND, added c5n.large to launch candidates list for instance i-00cce956596d3173b
2021/06/07 15:34:07 instance.go:495: MATCH FOUND, added c5n.xlarge to launch candidates list for instance i-00cce956596d3173b
2021/06/07 15:34:07 instance.go:495: MATCH FOUND, added i3.large to launch candidates list for instance i-00cce956596d3173b
2021/06/07 15:34:07 instance.go:495: MATCH FOUND, added m3.large to launch candidates list for instance i-00cce956596d3173b
2021/06/07 15:34:07 instance.go:495: MATCH FOUND, added m3.medium to launch candidates list for instance i-00cce956596d3173b
2021/06/07 15:34:07 instance.go:495: MATCH FOUND, added m3.xlarge to launch candidates list for instance i-00cce956596d3173b
2021/06/07 15:34:07 instance.go:495: MATCH FOUND, added m4.large to launch candidates list for instance i-00cce956596d3173b
2021/06/07 15:34:07 instance.go:495: MATCH FOUND, added m4.xlarge to launch candidates list for instance i-00cce956596d3173b
2021/06/07 15:34:07 instance.go:495: MATCH FOUND, added m5.large to launch candidates list for instance i-00cce956596d3173b
2021/06/07 15:34:07 instance.go:495: MATCH FOUND, added m5a.large to launch candidates list for instance i-00cce956596d3173b
2021/06/07 15:34:07 instance.go:495: MATCH FOUND, added m5ad.large to launch candidates list for instance i-00cce956596d3173b
2021/06/07 15:34:07 instance.go:495: MATCH FOUND, added m5d.large to launch candidates list for instance i-00cce956596d3173b
2021/06/07 15:34:07 instance.go:495: MATCH FOUND, added m5dn.large to launch candidates list for instance i-00cce956596d3173b
2021/06/07 15:34:07 instance.go:495: MATCH FOUND, added m5n.large to launch candidates list for instance i-00cce956596d3173b
2021/06/07 15:34:07 instance.go:495: MATCH FOUND, added r3.large to launch candidates list for instance i-00cce956596d3173b
2021/06/07 15:34:07 instance.go:495: MATCH FOUND, added r4.large to launch candidates list for instance i-00cce956596d3173b
2021/06/07 15:34:07 instance.go:495: MATCH FOUND, added r5.large to launch candidates list for instance i-00cce956596d3173b
2021/06/07 15:34:07 instance.go:495: MATCH FOUND, added r5a.large to launch candidates list for instance i-00cce956596d3173b
2021/06/07 15:34:07 instance.go:495: MATCH FOUND, added r5ad.large to launch candidates list for instance i-00cce956596d3173b
2021/06/07 15:34:07 instance.go:495: MATCH FOUND, added t3a.xlarge to launch candidates list for instance i-00cce956596d3173b
2021/06/07 15:34:07 instance.go:495: MATCH FOUND, added z1d.large to launch candidates list for instance i-00cce956596d3173b
2021/06/07 15:34:07 instance.go:577: Bidding base on demand price 0.067 to replace instance i-00cce956596d3173b
2021/06/07 15:34:07 instance.go:748: us-west-2 : Converting GP2 EBS volume to GP3 for new instance launched for staging-asg
2021/06/07 15:34:07 instance.go:748: us-west-2 : Converting GP2 EBS volume to GP3 for new instance launched for staging-asg
2021/06/07 15:34:07 region.go:526: us-west-2 Successfully deleted spot instance i-00cce956596d3173b launch event message from the SQS Queue https://sqs.us-east-1.amazonaws.com/255827576516/AutoSpotting.fifo
2021-06-07T11:34:07.625-04:00
runtime error: invalid memory address or nil pointer dereference: errorString
  [{
    "path": "github.com/aws/aws-lambda-go@v1.19.1/lambda/errors.go",
    "line": 33,
    "label": "lambdaPanicResponse"
  }, {
    "path": "github.com/aws/aws-lambda-go@v1.19.1/lambda/function.go",
    "line": 36,
    "label": "(*Function).Invoke.func1"
  }, {
    "path": "runtime/panic.go",
    "line": 965,
    "label": "gopanic"
  }, {
    "path": "runtime/panic.go",
    "line": 212,
    "label": "panicmem"
  }, {
    "path": "runtime/signal_unix.go",
    "line": 734,
    "label": "sigpanic"
  }, {
    "path": "/src/core/instance.go",
    "line": 705,
    "label": "convertLaunchConfigurationEBSVolumeType"
  }, {
    "path": "/src/core/instance.go",
    "line": 608,
    "label": "(*instance).convertLaunchConfigurationBlockDeviceMappings"
  }, {
    "path": "/src/core/instance.go",
    "line": 863,
    "label": "(*instance).processLaunchConfiguration"
  }, {
    "path": "/src/core/instance.go",
    "line": 945,
    "label": "(*instance).createRunInstancesInput"
  }, {
    "path": "/src/core/instance.go",
    "line": 534,
    "label": "(*instance).launchSpotReplacement"
  }, {
    "path": "/src/core/main.go",
    "line": 437,
    "label": "(*AutoSpotting).handleNewOnDemandInstanceLaunch"
  }, {
    "path": "/src/core/main.go",
    "line": 402,
    "label": "(*AutoSpotting).handleNewInstanceLaunch"
  }, {
    "path": "/src/core/main.go",
    "line": 220,
    "label": "(*AutoSpotting).processEventInstance"
  }, {
    "path": "/src/core/main.go",
    "line": 262,
    "label": "(*AutoSpotting).processEvent"
  }, {
    "path": "/src/core/main.go",
    "line": 285,
    "label": "(*AutoSpotting).EventHandler"
  }, {
    "path": "src/autospotting.go",
    "line": 55,
    "label": "eventHandler"
  }, {
    "path": "src/autospotting.go",
    "line": 74,
    "label": "Handler"
  }, {
    "path": "reflect/value.go",
    "line": 476,
    "label": "Value.call"
  }, {
    "path": "reflect/value.go",
    "line": 337,
    "label": "Value.Call"
  }, {
    "path": "github.com/aws/aws-lambda-go@v1.19.1/lambda/handler.go",
    "line": 124,
    "label": "NewHandler.func1"
  }, {
    "path": "github.com/aws/aws-lambda-go@v1.19.1/lambda/handler.go",
    "line": 24,
    "label": "lambdaHandler.Invoke"
  }, {
    "path": "github.com/aws/aws-lambda-go@v1.19.1/lambda/function.go",
    "line": 64,
    "label": "(*Function).Invoke"
  }, {
    "path": "reflect/value.go",
    "line": 476,
    "label": "Value.call"
  }, {
    "path": "reflect/value.go",
    "line": 337,
    "label": "Value.Call"
  }, {
    "path": "net/rpc/server.go",
    "line": 377,
    "label": "(*service).call"
  }, {
    "path": "runtime/asm_amd64.s",
    "line": 1371,
    "label": "goexit"
  }]

Cristian Magherusan-Stanciu · Answer 1 · Tue Jun 08 2021 22:29:28 GMT+0800 (China Standard Time)

Thanks for reporting this, I'll look into it within the next few days.

Cristian Magherusan-Stanciu · Answer 2 · Tue Jun 08 2021 22:44:19 GMT+0800 (China Standard Time)

Looks like we need a volume type, the code doesn't currently handle the situation in which the volume type is missing:

root_block_device {
  volume_size = "33"
}

Which volume type do you get for the running instances? That's probably coming from the AMI...

Cristian Magherusan-Stanciu · Answer 3 · Tue Jun 08 2021 23:03:31 GMT+0800 (China Standard Time)

@kulinsj I just pushed a change that should probably address this.

I don't currently have time to test it but please try the latest build and let me know if it helped. Otherwise I'll look into this in more detail towards the end of the week.

Jan · Answer 4 · Wed Jun 09 2021 02:59:08 GMT+0800 (China Standard Time)

Which volume type do you get for the running instances? That's probably coming from the AMI.

You're right, the AMI defines the EBS block type storage.

try the latest build and let me know if it helped

Sure did! All systems go now, spot instances are launching again. Thanks!

Cristian Magherusan-Stanciu · Answer 5 · Wed Jun 09 2021 03:56:19 GMT+0800 (China Standard Time)

Great to hear, thanks! Let me know if you run into any other issues

Later edit, I forgot to ask: doesn't AutoSpotting change the type of the EBS volume coming from the AMI? I would expect to have it changed from GP2 to GP3 as well

Jan · Answer 6 · Wed Jun 09 2021 04:14:13 GMT+0800 (China Standard Time)

In the logs it does show it being updated to GP3, yes

Cristian Magherusan-Stanciu · Answer 7 · Wed Jun 09 2021 09:06:25 GMT+0800 (China Standard Time)

Awesome, thanks for checking this!