aws-cloudformation / rain

A development workflow tool for working with AWS CloudFormation.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Handle throttling in cc deploy

ericzbeard opened this issue · comments

Services will throttle too many parallel requests to create resources. In cc deploy, all resources at the same level in the dependency graph get deployed at the same time, with no retries. We need to implement exponential backoff for RateExceeded or ThrottlingException errors, and batch requests if there are more than 5 going to same service at the same time.

rain forecast --include-iam permission checks can get throttled too:

$ rain forecast --experimental cfn.yaml --include-iam
FG002 FAIL on line 9: AWS::S3::Bucket Bucket - Insufficient permissions to create arn:aws:s3:::rain-XXX
	operation error IAM: SimulatePrincipalPolicy, exceeded maximum number of attempts, 3, https response error StatusCode: 400, RequestID: XXX, api error Throttling: Rate exceeded

rain forecast --include-iam permission checks can get throttled too:

Yeah that feature needs to be redesigned to take advantage of batching calls to the policy simulator, but the way the actions get batched are not consistent per resource, and not documented, so it's a trial and error approach.