peekaqueue

Peekaqueue provides a drop-in docker container for monitoring SQS queues and enabling queue-based auto-scaling of ECS services.

Unlike the default SQS metrics which are only published to CloudWatch every 5 minutes, peekaqueue can be configured to gather queue metrics much more frequently.

Features

Gets SQS queue metrics and exposes them via a Prometheus metrics endpoint
Optionally retrieves the desired/pending/running counts of queue consumers running in ECS
Optionally publishes the "backlog-per-instance" for your queue consumers to CloudWatch in order to configure auto-scaling

Prometheus metrics

Peekaqueue can be used as a Prometheus exporter. By default, the following SQS queue attributes are exposed as metrics at http://localhost:5000/metrics:

SQS Attribute	Metric Name	Labels
`ApproximateNumberOfMessages`	`sqs_available_messages_count`	`queue`
`ApproximateNumberOfMessagesDelayed`	`sqs_delayed_messages_count`	`queue`
`ApproximateNumberOfMessagesNotVisible`	`sqs_not_visible_messages_count`	`queue`

Monitoring Queue Consumers

One of the main goals of peekaqueue is to also monitor the consumers of SQS queues. It assumes that consumers are dockerized applications running in ECS. When configured, the following Prometheus metrics are exposed:

ECS Service Attribute	Metric Name	Labels
`DesiredCount`	`ecs_service_desired_count`	`ecs_cluster`, `ecs_service`
`PendingCount`	`ecs_service_pending_count`	`ecs_cluster`, `ecs_service`
`RunningCount`	`ecs_service_running_count`	`ecs_cluster`, `ecs_service`

Calculating Consumer Backlogs

To auto-scale queue consumers based on the number of messages, we need to determine the backlog-per-instance, that is how many messages are available per consumer instance (sqs_available_messages_count/ecs_service_running_count).

This value is exposed as the following Prometheus metric:

Metric Name	Labels
`ecs_service_backlog_count`	`ecs_cluster`, `ecs_service`, `queue`

In order to use this metric for auto-scaling it is published to CloudWatch as EcsServiceBacklog under the Custom metrics namespace (configurable). It includes the following CloudWatch metric dimensions:

Queue
EcsCluster
EcsService

Configuration

In most cases peekaqueue can be deployed direct to ECS using the images hosted on Docker Hub and configured using the following environment variables:

Environment Variable	Data Type	Default Value	Description
`PEEKAQ_Monitoring__IntervalInSeconds`	`int`	`15`	The interval in seconds in which peekaqueue will collect stats about your queues and consumers
`PEEKAQ_Monitoring__MetricsEndpointPort`	`int`	`5000`	The port that the prometheus metrics will run on
`PEEKAQ_Monitoring__MetricsEndpointPath`	`string`	`metrics/`	The path of the metrics endpoint. Must end in a trailing `/`
`PEEKAQ_Monitoring__CloudWatchNamespace`	`string`	`Custom`	The CloudWatch namespace for publishing metrics to
`AWS_Region`	`string`		The AWS region in which your resources reside e.g. `eu-west-2`
`PEEKAQ_Monitoring__Queues__X__Name`	`string`		The name of the queue you wish to monitor. `X` is the 0-based array indexer of the queues.
`PEEKAQ_Monitoring__Queues__X__ConsumerCluster`	`string`		The name of the ECS cluster in which your queue consumer resides
`PEEKAQ_Monitoring__Queues__X__ConsumerService`	`string`		The name of the ECS service in which your queue consumer resides
`PEEKAQ_Serilog__WriteTo__0__Args__restrictedToMinimumLevel`	`string`	`Information`	Configures the console logging level. It it recommended to set this to `Error` when running in production
`PEEKAQ_Serilog__WriteTo__1__Args__restrictedToMinimumLevel`	`string`	`Information`	If using [Seq](https://datalust.co/seq logging), configures the Seq logging level
`PEEKAQ_Serilog__WriteTo__1__Args__serverUrl`	`string`	`http://localhost:5341`	The Seq server URL
`PEEKAQ_Serilog__WriteTo__1__Args__apiKey`	`string`		API key for authenticating with Seq

If you are building the image from scratch you can instead update appsettings.json as required.

AWS Permissions and Credentials

It is assumed that you will run peekaqueue as an ECS service which should be given an ECS task role with the following permissions:

sqs:GetQueueUrl
sqs:GetQueueAttributes
cloudwatch:PutMetricData
ecs:DescribeServices

Below is an example policy that provides access to a single queue:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "sqs:GetQueueUrl",
                "sqs:GetQueueAttributes"
            ],
            "Resource": "arn:aws:sqs:eu-west-2:*:my-queue-name"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "cloudwatch:PutMetricData",
                "ecs:DescribeServices"
            ],
            "Resource": "*"
        }
    ]
}

Running Locally

To test peekaqueue on your local machine against AWS, you must provide your AWS access keys:

docker run --rm -it \
-e AWS_REGION=eu-west-2 \
-e AWS_ACCESS_KEY_ID=YOURKEYID \
-e AWS_SECRET_ACCESS_KEY=YOURKEY \
-e "PEEKAQ_MONITORING__QUEUES__0__Name=QUEUENAME" \
-p 5000:5000 benfoster/peekaqueue:latest

Developing peekaqueue

Peekqueue is a .NET Core 2.2 application. You can download the .NET Core runtime and tools here.

benfoster / peekaqueue