RequestLimitExceeded error when creating lambda functions inside VPC

Question

RequestLimitExceeded error when creating lambda functions inside VPC

zacharynevin opened this issue 7 years ago · comments

Zachary Nevin commented 7 years ago

This is a Bug Report

Description

What went wrong?
I have around 40 lambda functions in my serverless file. When I do serverless deploy without a VPC configuration, it deploys fine.

However, when I add a VPC configuration like this:

vpc:
  securityGroupIds:
    - ***
  subnetIds:
    - ***

I have a few of the functions create and then the CF deployment fails because of the following error:

  Serverless Error ---------------------------------------
 
     An error occurred while provisioning your stack: MyLambdaFunction
     - Your request has been throttled by EC2, please make
     sure you have enough API rate limit. EC2 Error Code:
     RequestLimitExceeded. EC2 Error Message: Request limit
     exceeded..
 
  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
 
  Your Environment Information -----------------------------
     OS:                 darwin
     Node Version:       7.6.0
     Serverless Version: 1.8.0

After filing an AWS support request, this error appears because there is a hard limit on the create function. This can be solved in one of two ways:

Split the serverless.yaml file into smaller pieces.
Add the DependsOn parameter to each lambda function.

Personally I like the 2nd option because I won't need to modify my existing serverless.yaml file. However, I could not find anything in the Serverless documentation or the Github issues list that indicates there is an optional DependsOn property for lambda functions, so that I can delay the creation of a lambda function until another one has finished creating.

If this exists, can you let me know how it works? If it does not exist, is there is a package or a feature proposal for this?

Thanks so much!

Zachary Nevin · Answer 1 · Thu Mar 09 2017 21:45:03 GMT+0800 (China Standard Time)

@pmuens What would be your recommendation for avoiding this error, for now?

Philipp Muens · Answer 2 · Thu Mar 09 2017 23:25:17 GMT+0800 (China Standard Time)

Hey @zacharynevin you could use the resources section to overwrite the generated CloudFormation code and extend the Lambda code: https://serverless.com/framework/docs/providers/aws/guide/resources

Other than that you could ask AWS to increase your limits.

Zachary Nevin · Answer 3 · Fri Mar 10 2017 14:42:23 GMT+0800 (China Standard Time)

@pmuens it worked! For other interested parties, this is what I did:

service: myservice
provider:
  type: aws
  ...

functions:
  myExampleFunction:
    handler: ...
  my-other-example-function:
    handler: ...

resources:
  Resources:
    # notice how the string is always PascalCase, and the resource is appended with "LambdaFunction"
    MyExampleFunctionLambdaFunction:
      Type: AWS::Lambda::Function
      DependsOn: MyOtherExampleFunctionLambdaFunction

Philipp Muens · Answer 4 · Fri Mar 10 2017 18:04:33 GMT+0800 (China Standard Time)

Great to hear that @zacharynevin 🎉

Thanks for sharing your config!

Brian Winant · Answer 5 · Sun Mar 25 2018 12:23:24 GMT+0800 (China Standard Time)

Having also experienced this issue, and not wanting to have to manually overwrite the resources section for each lambda, I created a plugin that automatically does it: https://github.com/bwinant/serverless-dependson-plugin

Nathan Allan · Answer 6 · Wed Apr 25 2018 06:06:31 GMT+0800 (China Standard Time)

Why was this closed? The "solution" above is at best a work-around.

Rusty Welch · Answer 7 · Thu May 30 2019 15:35:17 GMT+0800 (China Standard Time)

For anyone hitting this, as stated above, requesting an increased limit will solve this (but I wanted it to work now). Setting my createNatGateway to 2 in the serverless.yaml's vpcConfig section (I'm using us-west-2 region, for which there's 4 availability zones (2a, 2b, 2c and 2d) as of this date) and adding in zones:

    zones:
      - us-west-2a
      - us-west-2b

allowed me to deploy successfully. I also refactored my lambdas (I'm a newbie, a co-worker provided the initial refactor) to use one two functions instead of four by abstracting 3 of them into one 'main' function that then executes the relevant code, which has helped as well.
I haven't had to use the DependsOn, but I am interested if anyone has used bwinant's plugin and could report back - might be interesting to see if there's much timing difference when setting it to 2, as that would cut the number significantly. Cheers -

Vicary A. · Answer 8 · Mon Aug 31 2020 14:08:42 GMT+0800 (China Standard Time)

@RangerRusty Sorry to bring this up again, but this hit us hard. We have >100 functions. What exactly is the name of the limit that I should be requesting?

Ross Coundon · Answer 9 · Thu Apr 01 2021 01:47:51 GMT+0800 (China Standard Time)

@RangerRusty Sorry to bring this up again, but this hit us hard. We have >100 functions. What exactly is the name of the limit that I should be requesting?

It's been a while since you asked @vicary - did you find out what the limit was called that you needed to increase?

Hines Vaughan · Answer 10 · Thu Apr 01 2021 02:09:51 GMT+0800 (China Standard Time)

@rcoundon We ended up using DependsOn to slow everything down. Forcing each lambda to depend on the one before it so that CF only creates one at a time. We only do this when changing a setting that affects a bunch of lambdas. Once the change is deployed, we can remove the DependsOn and build normally.

We only allow 1 lambda per nested stack to update at a time but 2 of our nested stacks run at the same time so it is technically updating 2 lambdas at any one time.

Very frustrating that this is necessary. I never could find the rate limit that needed to be increased.

Ross Coundon · Answer 11 · Thu Apr 01 2021 03:22:00 GMT+0800 (China Standard Time)

@hvaughan3 - thanks for letting me know. For the time being, I'm using the serverless-dependson-plugin which allows configuration of how many parallel batches can run. It's set to three for now and seems to be behaving.

Vicary A. · Answer 12 · Fri Apr 09 2021 17:07:04 GMT+0800 (China Standard Time)

@RangerRusty Sorry to bring this up again, but this hit us hard. We have >100 functions. What exactly is the name of the limit that I should be requesting?

It's been a while since you asked @vicary - did you find out what the limit was called that you needed to increase?

@rcoundon Read the Non-mutating actions row in API throttling limits, they don't even bother listing all of the actions there. I am imagining 100+ of those methods.

My team is subscribing to paid support plans, their response at least nailed down to some undocumented internal limits under the above calls, which is not meant to be raised in normal circumstances. Further responses from them are vague, I guess they are not supposed to tell how those internal limits works.

My only way to safely deploy a stack is to stick with the lowest related Bucket Refill Rate in the table above.

Ross Coundon · Answer 13 · Fri Apr 09 2021 17:10:12 GMT+0800 (China Standard Time)

@vicary Thanks for getting back to me. Yeah, frustratingly opaque. We're getting around it with some chaining using dependsOn which slows things down but it's manageable right now. Thanks again

Kyle Lynch-Kurzawa · Answer 14 · Fri Jul 26 2024 16:20:56 GMT+0800 (China Standard Time)

For anyone still experiencing this issue -

After reaching out to AWS support the limit appears to be 15 lambdas per second (which cannot be increased according to them). Their recommendation at this time is to use the DependsOn attribute of lambdas in the CF template to stagger / group the deployment of lambdas in a more sequential manner instead of them all being updated in parallel.

So @zacharynevin 's solution is the recommended one

Resources:

The example I was provided:

========EXAMPLE DEPENDENCY SETUP========
- Function 1
- Function 2 (DependsOn Function1)
- Function 3 (DependsOn Function1)
- Function 4 (DependsOn Function1)
- Function 5 (DependsOn Function1)
- Function 6 (DependsOn Function1)
- Function 7 (DependsOn Function1)
- Function 8 (DependsOn Function7)
- Function 9 (DependsOn Function8)
- Function 10 (DependsOn Function8)
- Function 11 (DependsOn Function8)
- Function 12 (DependsOn Function8)
- Function 13 (DependsOn Function8)
- Function 14 (DependsOn Function8)
- Function 15 (DependsOn Function14)
- Function 16 (DependsOn Function15)
- Function 17 (DependsOn Function15)
- Function 18 (DependsOn Function15)
- Function 19 (DependsOn Function15)
- Function 20 (DependsOn Function15)
- Function 21 (DependsOn Function15)
- Function 22 (DependsOn Function21)
- Function 23 (DependsOn Function22)
- Function 24 (DependsOn Function22)
- Function 25 (DependsOn Function22)
- Function 26 (DependsOn Function22)
- Function 27 (DependsOn Function22)
- Function 28 (DependsOn Function22)
- Function 29 (DependsOn Function28)
- Function 30 (DependsOn Function29)
- Function 31 (DependsOn Function29)
- Function 32 (DependsOn Function29)
- Function 33 (DependsOn Function29)
- Function 34 (DependsOn Function29)
- Function 35 (DependsOn Function29)
- Function 36 (DependsOn Function35)
- Function 37 (DependsOn Function36)
- Function 38 (DependsOn Function36)
- Function 39 (DependsOn Function36)
- Function 40 (DependsOn Function36)
- Function 41 (DependsOn Function36)
- Function 42 (DependsOn Function36)
========EXAMPLE DEPENDENCY SETUP========