Handle throttling in cc deploy
ericzbeard opened this issue · comments
Services will throttle too many parallel requests to create resources. In cc deploy
, all resources at the same level in the dependency graph get deployed at the same time, with no retries. We need to implement exponential backoff for RateExceeded
or ThrottlingException
errors, and batch requests if there are more than 5 going to same service at the same time.
rain forecast --include-iam
permission checks can get throttled too:
$ rain forecast --experimental cfn.yaml --include-iam
FG002 FAIL on line 9: AWS::S3::Bucket Bucket - Insufficient permissions to create arn:aws:s3:::rain-XXX
operation error IAM: SimulatePrincipalPolicy, exceeded maximum number of attempts, 3, https response error StatusCode: 400, RequestID: XXX, api error Throttling: Rate exceeded
rain forecast --include-iam
permission checks can get throttled too:
Yeah that feature needs to be redesigned to take advantage of batching calls to the policy simulator, but the way the actions get batched are not consistent per resource, and not documented, so it's a trial and error approach.