sbstjn / serverless-dynamodb-autoscaling

Serverless Plugin for Amazon DynamoDB Auto Scaling configuration.

Home Page:https://sbstjn.com/serverless-dynamodb-auto-scaling-with-cloudformation.html

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Rate Exceeded error when adding scaling to preexisting stack.

cdichiara opened this issue · comments

Hi,

Nice plugin, by the way ... really appreciate that you made it.

I think Issue #3 might still be a problem for preexisting stacks where scaling is added:
https://forum.serverless.com/t/autoscaling-for-amazon-dynamodb/2071/15?u=cdichiara
I suspect that you don't have prolific enough dependencies to protect against updates to preexisting stacks:

          "irhbackendAutoScalingTargetWriteDatabaseStageRegion": {
            "DependsOn": [
              "UsersDb",
              "irhbackendDynamoDBAutoscaleRoleDatabaseStageRegion"
            ],

These are logical dependencies ... to protect against rate exceeding I think you're going to want illogical dependencies that serialize the creation of scaling policies and scaling targets, or whatever's adding up to whatever limit it's triggering. So, in other words, something like this:

          "irhbackendAutoScalingTargetWriteDatabaseStageRegion": {
            "DependsOn": [
              "UsersDb",
              "irhbackendDynamoDBAutoscaleRoleDatabaseStageRegion",
              "irhbackendAutoScalingTargetWriteLastDatabaseStageRegion"
            ],

It works for you when you do the initial install because the creations must be serialized by the logical dependencies, somehow.

I'll try to get around to a possible fix in a pull request, but I can't do it now ... my workaround is basically adding autoscaling to one database at a time, up to now it hasn't triggered the rate issue, but it's ponderous.

Thanks!

After manually adding auto-scaling to my existing tables and hitting the rate limit problem, I can confirm the current DependsOn implementation in the plugin will not limit the rate properly and will get Rate exceeded when adding or changing many AWS::ApplicationAutoScaling::ScalingPolicy resources when the tables already exist and may also get it if many ScalingPolicy are changed in the same update.

DependsOn is the proper solution, but what it depends on is wrong. In order to eliminate the Rate exceeded error in all cases, the DependsOn for AWS::ApplicationAutoScaling::ScalingPolicy should be the previous ScalingPolicy and nothing else. This will serialize additions and changes to the policies when creating or changing them regardless if the tables were previously created or not.

This solution is similar to how AWS explains that to avoid table related rate limits, place DependsOn on your table to depend on the previous table.