abhisheksp / kinesis-scaling-workflow

Simple Step Function to scale up/down your Kinesis Data Stream shard to the desired shard count.

Kinesis Scaling Workflow

Simple Step Function to scale up/down your Kinesis Data Stream shard to the desired shard count.

Why?

Current Limitation of one single Kinesis Shard Update operation:

• Cannot scale up to more than double your current shard count for a stream
• Cannot scale down below half your current shard count for a stream

In order to scale up or scale down your shards beyond the double/half thresholds, you need to perform multiple update shard operations and wait in between. This package offers a simple step function to automate these steps.

Ref: Update Shard Count Doc

PS: I also wanted to try out the relatively new AWS SDK integrations from AWS Step Functions[Ref: Announcement]

CDK Deployment Steps

This packages uses AWS CDK to define and deploy infrastructure which includes a Lambda and a Step Function State Machine.

Pre-requisites: Install CDK and Configure AWS Credentials[Ref]

cd cdk
cdk synth
cdk deploy

Usage

After deployment, you can execute the State Machine via AWS console or via AWS CLI

State Machine Input Example:

{
  "stream_name": "test-stream",
  "desired_shard_count": 3
}

AWS CLI execution examole

aws stepfunctions start-execution \
--state-machine-arn arn:aws:states:us-east-1:123456789:stateMachine:KinesisScalingWorkflow \
--input "{\"stream_name\": \"test-stream\", \"desired_shard_count\": 1}"

Removal

cd cdk
cdk destroy

Manually remove any retained Lambda CloudWatch Log Groups