Create a function to process Cloudwatch logs streams to detect errors.
π§ This is a work in progress.
Create two lambda resources:
- For regular workloads - logs to a specific cloudwatch log group
- For supplementary workloads - processing the logs from #1
- Setup AWS SSM for storing senstive info (slack webhook url)
- Setup SNS / slack notification for errors [BONUS]
Draft of the architecture (v1).
- Scaffold infra for print log lambda and error processor lambda
- Scaffold send slack webhook alert
- Setup AWS SSM
- Setup SQS to delay processing of error notification events
- Setup other lambdas notification services (email, sms)
- Setup lambda to publish to sns notifier topic
- Setup SNS and SQS infrastructure for fanout
- Setup SNS for notification channels (ex. text, slack and email)
- Include architecture diagram
β οΈ Important: Remember to destroy all resources after finish testing
terraform destroy -auto-approve
-
Allow messages to be sent to your slack channel, see Slack Instructions.
-
Add AWS secret and key to your environment (or use template below and fill in envs)
# setup-env.sh
export AWS_ACCESS_KEY_ID=<xxxx>
export AWS_SECRET_ACCESS_KEY=<xxxx>
export AWS_DEFAULT_REGION=us-east-1
export TF_VAR_s3_bucket_name=<unique-s3-bucket-name>
export TF_VAR_slack_webhook_endpoint_url=<slack-web-hook-url>
. ./setup-env.sh
- Build assets
Create a new version of the assets to be shipped and deployed π.
Command:
yarn version:patch
What you should see:
./deploy/
βββ lambda-process-cw-logs-<version>.zip
- Invoking Lambda (using
aws-cli
)
This will kick off the trigger to start logging error in our print log error lambda.
aws lambda invoke \
--function-name process-cw-logs-print-log-lambda \
response.json
-
Inspect the message in your slack channel
-
Inspect Cloudwatch Logs (if you don't see message, there may be issues)
To debug, please enable debug mode by uncommenting the following line:
resource "aws_lambda_function" "error_processing_lambda" {
...
environment {
variables = {
...
#SLACK_MS_DEBUG = true <-- uncomment
}
}
}
- Node.js (12.x)
- Terraform (12.x)
- Jest (26.x)
- Typescript (3.9.7)
- rollup (1.3.x)
- axios (^0.21.0)
- @speee-js/jsx-slack (^2.6.x)
When defining the lambda permission for CloudWatch logs to invoke functions.
Ensure the Principal
field is correct and has the right format:
resource "aws_lambda_permission" "allow_cloudwatch_logs" {
...
principal = "logs.<aws-region>.amazonaws.com"
}
Otherwise, you may run into an error when running terraform:
This section contains the helpful links for this particular task / project.
Terraform:
- Terraform, Resource: aws_cloudwatch_log_subscription_filter
- Terraform, Resource: aws_cloudwatch_log_group
- Terraform, Resource: aws_lambda_permission
AWS:
Stackoverflow: