sbstjn / serverless-dynamodb-autoscaling

Serverless Plugin for Amazon DynamoDB Auto Scaling configuration.

Home Page:https://sbstjn.com/serverless-dynamodb-auto-scaling-with-cloudformation.html

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

The security token included in the request is invalid...

cdimitroulas opened this issue · comments

When I first deployed using this plugin I did not encounter any issues. But after removing a test deployment and re-deploying it later in the day I encountered an error which was persistent even after a couple of retries

My deployments started failing with an error along the lines of:
Unable to retrieve the capacity for resource: table/<tablename>, scalable dimention: dynamodb:table:WriteCapacityUnits. Reason: The security token included in the request is invalid...

I see in this thread that there was a discussion about the same problem but this was quite some time ago and there have been several releases since then.

Is this an error that was already solved? It would be good to know whether this is an error in my configuration of if this is a known problem with this plugin.

For the moment I've remove the autoscaling from my configuration to get the deployment to work, but I would love to use this great plugin :D

I think this error is somewhere in the Serverless framework or how the CloudFormation resources are handled. Basically the plugin extends the CloudFormation resources created by Serverless by a few items. if you have any clue how to prevent this, I would love to implement them :)

I've approached same error, and it was fully reproducible on my side when I removed the stack and tried to redeploy it (create it) right afterwards.

I contacted AWS support, and was told that's the race condition issue, that can't be fixed with DependsOn attribute, and at this point the only solution is to in first pass deploy stack just with IAM role and in second pass deploy autoscaling resources.

In serverless-plugin-dynamodb-autoscaling I've introduced iamOnly option that helps handling that. Possibly something similar can be provided in this plugin.

In case someone is looking for a workaround, I was able to get autoscaling working via custom resources:

ScalingRole:
  Type: "AWS::IAM::Role"
  Properties:
    AssumeRolePolicyDocument:
      Version: "2012-10-17"
      Statement:
        -
          Effect: "Allow"
          Principal:
            Service:
              - application-autoscaling.amazonaws.com
          Action:
            - "sts:AssumeRole"
    Path: "/"
    Policies:
      -
        PolicyName: "root"
        PolicyDocument:
          Version: "2012-10-17"
          Statement:
            -
              Effect: "Allow"
              Action:
                - "dynamodb:DescribeTable"
                - "dynamodb:UpdateTable"
                - "cloudwatch:PutMetricAlarm"
                - "cloudwatch:DescribeAlarms"
                - "cloudwatch:GetMetricStatistics"
                - "cloudwatch:SetAlarmState"
                - "cloudwatch:DeleteAlarms"
              Resource: "*"
ReadCapacityScalableTarget:
  Type: "AWS::ApplicationAutoScaling::ScalableTarget"
  Properties:
    MaxCapacity: 20
    MinCapacity: 5
    ResourceId: table/comments
    RoleARN:
      Fn::GetAtt: [ ScalingRole, Arn ]
    ScalableDimension: dynamodb:table:ReadCapacityUnits
    ServiceNamespace: dynamodb 
ReadScalingPolicy:
  Type: "AWS::ApplicationAutoScaling::ScalingPolicy"
  Properties:
    PolicyName: ReadAutoScalingPolicy
    PolicyType: TargetTrackingScaling
    ScalingTargetId:
      Ref: CommentsReadCapacityScalableTarget
    TargetTrackingScalingPolicyConfiguration:
      TargetValue: 80.0
      ScaleInCooldown: 60
      ScaleOutCooldown: 60
      PredefinedMetricSpecification:
        PredefinedMetricType: DynamoDBReadCapacityUtilization

Also I did some tests of how this works: https://60devs.com/testing-dynamodb-automatic-scaling.html

In case someone is looking for a workaround, I was able to get autoscaling working via custom resources

It's exactly what this plugin does, and in most cases it works, there's no doubts.

Still, the problem this issue describes is likely a result of rare race condition issue that's not addressed as expected on CloudFormation side. It's exposed just in some cases (e.g. deployment of stack followed immediately by its removal)

I'm experiencing this as well using the workaround listed above where all my auto scaling is setup using the cloudformation scalable target/policy. I've been experiencing this over and over to the point, I can't even reliably deploy my stack to AWS. I'm going to try doing as medikoo described and waiting for awhile between removing and redeploying as I've run into this error five times in a row.

Unable to retrieve capacity for resource: table/xyz, scalable dimension: dynamodb:table:ReadCapacityUnits. Reason: The security token included in the request is invalid.

Sometimes it will give this error for the table scaling and sometimes for the indexes. Right now, this error was generated from the first stack that is deployed which sets up the dynamodb table and applies auto scaling to read/write, which is prior to any indexes having auto scaling attached.

Waiting in between removing the stack and redeploying avoided this error from popping up. I was able to deploy 2 tables with autoscaling and 10 indexes with auto scaling. So even on the other stacks where the indexes were having autoscaling attached it worked without issue.

I'll come back and update if waiting in between removal and redeploy doesn't work in the future.

Closing due to inactivity