Name: cloudwatch-alarm
Description: Amazon CloudWatch Alarm 2.0. CloudWatch to Slack smart notification
GitHub: https://github.com/korniichuk/cloudwatch-alarm
- Introduction
- How it works
- Requirements
- Python lib versions
- Create Slack webhook
- Create Amazon SNS topic
- Create Amazon CloudWatch Alarm
- Create Amazon S3 bucket
- Create Amazon Lambda function
- Edit AWS IAM role
- Test Amazon Lambda function
- Change Slack message retention
Please, install Python packages:
$ pip install -t . -r requirements.txt
Navigate to https://<your-team-domain>.slack.com/apps, like https://example.slack.com/apps. Search for and select Incoming WebHooks. Click Add Configuration button:
Choose the default channel where messages will be sent (like #example) and click Add Incoming WebHooks integration. Copy and save the webhook URL (like https://hooks.slack.com/services/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) from the setup instructions.
Note: You need admin rights to your Slack.
Note: Slack marks public channels with # (e.g. #public) and private channels without # (e.g. private).
Create new Amazon SNS topic: https://docs.aws.amazon.com/sns/latest/dg/sns-getting-started.html#CreateTopic. Like example topic with arn:aws:sns:eu-west-1:539199393808:example ARN.
Navigate to https://console.aws.amazon.com/cloudwatch/ and create new Amazon CloudWatch Alarm. Click Select metric button:
Select metric, like EC2 -> Per-Instance Metrics -> CPUUtilization:
Choose Source tab and select Image API checkbox. Copy and save json code. Click Select metric button:
Saved json code example:
{
"metrics": [
[ "AWS/EC2", "CPUUtilization", "InstanceId", "i-0d91fdf2dbb765977", { "period": 300, "stat": "Average" } ]
],
"title": "The percentage of CPU utilization",
"start": "-PT3H",
"end": "P0D",
"timezone": "+0100"
}
Set up Alarm details. Enter Name (e.g. CPUUtilization) and Description (e.g. The percentage of CPU utilization). Specify the alarm condition. For example: select >=, enter 80, and enter 3 datapoints.
Connect CloudWatch Alarm to created SNS topic and finally click Create Alarm button:
AWS docs: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html
Navigate to https://console.aws.amazon.com/s3/ and create new S3 bucket (e.g. example). Create new lifecyle rule for bucket (e.g. DeleteTmpAfter72h) with tmp/ prefix filter:
Configure expiration as below. Expire after 3 days. Permanently delete after 3 days. Clean up incomplete multipart uploads after 1 day.
Edit public access settings to your S3 bucket. Select your S3 bucket (e.g. example) and click Edit public access settings button. Deselect all checkboxes and click Save button:
Enter confirm to confirm field and click Confirm button:
Navigate to https://console.aws.amazon.com/lambda/. Click Create function. Click Blueprints. Search for and select cloudwatch-alarm-to-slack-python3. Click Configure button:
Enter function name (e.g. example) to Name field. In Role select Create a new role from one or more templates. In Role name field enter role name (e.g. example).
Go to SNS trigger section and select your SNS topic. For example: example topic with arn:aws:sns:eu-west-1:539199393808:example ARN. Select the Enable trigger checkbox.
Create KMS Key or use an existing KMS Key. For example: arn:aws:kms:eu-west-1:539199393808:key/4c3126f2-f3bf-453e-b32d-58c9356d84b7.
Go to Lambda function code section. Go to Encryption configuration subsection and select the Enable helpers for encryption in transit checkbox. Select AWS KMS key to encrypt in transit (e.g. arn:aws:kms:eu-west-1:539199393808:key/4c3126f2-f3bf-453e-b32d-58c9356d84b7).
Paste Slack channel into the slackChannel environment variable, like example. The Slack channel does not contain private info, so do NOT click Encrypt button.
Paste Slack webhook URL into the kmsEncryptedHookUrl environment variable. You must exclude the protocol from the URL (e.g. hooks.slack.com/services/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). Click Encrypt button. Finally click Create function button:
Clone cloudwatch-alarm repository:
$ git clone https://github.com/korniichuk/cloudwatch-alarm.git
Modify cloudwatch-alarm/lambda_function.py file. Replace confluence_url and cloudwatch_url var values. For example, from:
confluence_url = 'http://www.korniichuk.com'
cloudwatch_url = 'http://www.korniichuk.com'
to:
confluence_url = 'https://bit.ly/1d3LdqJ'
cloudwatch_url = 'https://amzn.to/2NeZooo'
Update metric var value with saved Image API json. For example, from:
metric = """{
"metrics": [[
"LogMetrics",
"MetricName",
{"period": 3600, "stat": "Sum"}]],
"title": "Title",
"start": "-P1D",
"end": "P0D",
"timezone": "+0100"}"""
to:
metric = """{
"metrics": [[
"AWS/EC2",
"CPUUtilization",
"InstanceId",
"i-0d91fdf2dbb765977",
{"period": 300, "stat": "Average"}]],
"title": "The percentage of CPU utilization",
"start": "-PT3H",
"end": "P0D",
"timezone": "+0100"}"""
Docs: Metric widget structure and syntax
Replace bucket_name var value to your bucket name. For example, from:
bucket_name = 'korniichuk'
to:
bucket_name = 'example'
Navigate to cloned cloudwatch-alarm repository and install Python packages:
$ cd cloudwatch-alarm
$ pip install -t . -r requirements.txt
Prepare zip.zip with lambda_function.py file and Python dependencies:
$ rm -r *.dist-info __pycache__
$ zip -r zip.zip .
Navigate to your lambda function (e.g. example). Go to Function code section and select Upload a .zip file from Code entry type dropdown. Click Upload button. Upload zip.zip file. Finally click Save button:
Note: We use boto3 ver. 1.9.41 and botocore ver. 1.12.41 because default Lambda's packages do not support get_metric_widget_image function.
Navigate to https://console.aws.amazon.com/iam/. Select your role (e.g. example). Select Permissions tab. Click Attach policies button:
Search for and select AmazonS3FullAccess and CloudWatchFullAccess. Click Attach policy button. See final result below.
Note: Do not use *FullAccess roles on production. You can grant access to concrete S3 bucket (e.g. example). You can precisize CloudWatchFullAccess policy to cloudwatch:* plus logs:* only.
Navigate to your lambda function (e.g. example). Click Select a test event.. and select Configure test events:
Enter event name to Event name field (e.g. Example). Copy json code from test-event.json file to code area or enter your own. Click Create button:
Select created event (e.g. Example) and click Test button:
Note: You can maximize CPU utilization with $ dd if=/dev/zero of=/dev/null command.
Navigate to Slack channel. Click Channel Settings icon. Select Edit message retention...:
Change default message retention to 3 Days. Click Save button:
