Giters

StateFarmIns / terraform-aws-default-log-retention

A simple Terraform module that sets log retention policies for all CloudWatch Log Groups so that you don't have to!

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Introduction

terraform-aws-default-log-retention is a Terraform module that will set default retention periods on all CloudWatch Log Groups in a region.

By default, AWS does not set a default retention on Log Groups, meaning that any logs emitted today are kept forever. This module aims to solve that in an automated fashion, rather than requiring you to set retention on all Log Groups yourself.

Prerequisites

CloudTrail must be logging to a bucket in the account, otherwise the event-driven Lambda will not run. The global Lambda will still pick up events.

See Terraform documentation for creating a basic CloudTrail.

Detail

It behaves in 2 ways:

  • A one-time run which sets retention for all existing log groups ("Global Retention Setter" in diagram below)
  • Event-driven based on CloudWatch Log Group creation events. Sets retention on the group.

If log_group_tags is set, the module will also set a retention=Set by AWS Default Log Retention project. tag on any group it touches, to make the changes seem less "magical" to end-users.

Notes

  • Neither Lambda will override existing retention schedules (non-zero) set by product teams.
  • Neither Lambda will override groups that already have a retention tag

Diagram

diagram

Example Usage

Single Region

locals {
 notification_list               = ["your.email@example.com"]
 global_log_retention_run_period = 60 * 12 # Twice a day
 log_retention_in_days           = 30 # See tf-inputs.tf for allowed values
 log_retention_name              = "default-log-retention"
}

module "log_retention" {
  source = "git::https://github.com/StateFarmIns/terraform-aws-default-log-retention.git"

  name = local.log_retention_name

  log_level                       = "info"
  log_retention_in_days           = local.log_retention_in_days
  global_log_retention_run_period = local.global_log_retention_run_period

  alarm_configuration = {
    email_notification_list = local.notification_list
  }
}

Multi-region

# Define an aws provider for your primary region, and an aliased "secondary" provider for the secondary region

locals {
 notification_list               = ["your.email@example.com"]
 global_log_retention_run_period = 60 * 12 # Twice a day
 log_retention_in_days           = 30 # See tf-inputs.tf for allowed values
 log_retention_name              = "default-log-retention"
}

module "log_retention" {
  source = "git::https://github.com/StateFarmIns/terraform-aws-default-log-retention.git"

  name = local.log_retention_name

  log_level                       = "info"
  log_retention_in_days           = local.log_retention_in_days
  global_log_retention_run_period = local.global_log_retention_run_period

  alarm_configuration = {
    email_notification_list = local.notification_list
  }
}

module "log_retention_secondary" {
  source = "git::https://github.com/StateFarmIns/terraform-aws-default-log-retention.git"

  providers = {
    aws = aws.secondary
  }

   name = local.log_retention_name

  log_level                       = "info"
  log_retention_in_days           = local.log_retention_in_days
  global_log_retention_run_period = local.global_log_retention_run_period

  alarm_configuration = {
    email_notification_list = local.notification_list
  }
  
  iam_role_suffix = "-secondary" # Required to prevent IAM role names from conflicting
}

The module assumes that you are using AWS provider default tags, which implicitly adds all of your common tags to all resources. If you can't use default_tags, pass in the tags variable to the module.

CloudWatch Metrics

This module emits CloudWatch Metrics. The metric namespace can be overridden by using the metric_namespace variable. The default metric namespace is LogRetention.

Metric Name Useful Statistic Emitted by Global Lambda Emitted by Event Lambda Description
Total Sum ✅ ❌ Total number of CloudWatch Log Groups present in the region.
Updated Sum ✅ ✅ Number of CloudWatch Log Groups updated with retention by the module.
AlreadyHasRetention Sum ✅ ✅ Number of CloudWatch Log Groups that have a nonzero retention value already.
AlreadyTaggedWithRetention Sum ✅ ✅ Number of CloudWatch Log Groups that have 0 retention set but are tagged with retention. These groups are ignored by the module.
Errored Sum ✅ ❌ Number of CloudWatch Log Group updates which encountered an error. The Global setter will report errors and keep working on the remaining groups.

Requirements

No requirements.

Providers

Name Version
archive 2.4.0
aws 5.4.0

Modules

No modules.

Resources

Name Type
aws_cloudwatch_event_rule.global_log_retention resource
aws_cloudwatch_event_rule.log_group_creation resource
aws_cloudwatch_event_target.global_log_retention resource
aws_cloudwatch_event_target.log_group_creation resource
aws_cloudwatch_log_group.global_log_retention_lambda resource
aws_cloudwatch_log_group.log_retention_lambda resource
aws_cloudwatch_metric_alarm.alarm resource
aws_iam_role.log_retention resource
aws_lambda_function.global_log_retention resource
aws_lambda_function.log_retention resource
aws_lambda_function_event_invoke_config.global_log_retention resource
aws_lambda_function_event_invoke_config.log_retention resource
aws_lambda_invocation.run_on_existing_groups resource
aws_lambda_permission.global_log_retention resource
aws_lambda_permission.log_retention resource
aws_sns_topic.alarms resource
aws_sns_topic_policy.alarms resource
aws_sns_topic_subscription.alarms resource
archive_file.global_log_retention data source
archive_file.log_retention data source
aws_caller_identity.current data source
aws_iam_account_alias.current data source
aws_iam_policy_document.alarms data source
aws_iam_policy_document.lambda_assume data source
aws_iam_policy_document.log_retention data source
aws_iam_session_context.current data source
aws_region.current data source

Inputs

Name Description Type Default Required
alarm_configuration Provide either sns_topic_arn to an existing SNS topic, or a list of email users email_notification_list to subscribe for notifications. Passing null or omitting the argument turns off alarms. Note that retention setting is retried automatically, so an alarm may mean that it failed the first time and succeeded the second time. Investigating logs for each failure is recommended. any null no
global_log_retention_run_period Set to a number of minutes to invoke the global log retention Lambda on a schedule. Note that running it may cause perpetual diffs in other people's Terraform if they are creating a log group and not setting retention. number 360 no
https_egress_security_group_id If using a VPC, pass the ID of a security group which provides egress on port 443 to CloudWatch Logs. string null no
iam_role_suffix Due to Terraform limitations, this module always creates an IAM role. Pass in a suffix for the IAM role name so that it does not conflict between regions. string "" no
kms_key_arn If using a KMS key, provide it. string null no
log_group_tags Set of tags to put on all log groups when retention is set. If not set, no tags will be added. If set, a retention tag will automatically be added to this list. map(string) null no
log_level Override Lambda log level (trace/debug/info/warn/error) string "info" no
log_retention_in_days Default number of days to set on new log groups. Must be a valid option that CloudWatch Logs support: https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_PutRetentionPolicy.html#API_PutRetentionPolicy_RequestParameters number 90 no
metric_namespace CloudWatch Metric namespace for custom metrics emitted by these Lambdas. string "LogRetention" no
name Base name for all resources. E.x. . string n/a yes
permissions_boundary_arn Provide a permissions boundary ARN if you are bound by one. string null no
set_on_all_existing_groups Set to false to disable running a bit of code which will set retention on all existing groups. bool true no
subnet_ids If using a VPC, provide the IDs of the subnets you would like to deploy the Lambda to. list(string) null no
tags Adds tags to all created resources. It is highly recommended to use the AWS Provider's default tags instead of this variable. See: https://www.hashicorp.com/blog/default-tags-in-the-terraform-aws-provider. You can also use this input to add additional tags above and beyond the tags that are added by default_tags. map(string) null no

Outputs

No outputs.

Local development

Installations (other than Rust/cargo):

  • brew install zig
  • cargo install cargo-lambda
  • cargo install cargo-insta
  • rustup component add rust-src (optional; for VSCode Rust Analyzer)
  • VSCode Rust Analyzer extension (optional)

Building

./scripts/build-release.sh

Running Code Coverage

TODO: Fix Tarpaulin

Tarpaulin is a tool for coverage but it only runs on Linux. Furthermore it only runs when certain security features are disabled. It is not currently automated, so coverage will only be updated manually by hand. Feel free to check in the reports/* folder reports that Tarpaulin creates so that we at least have some sort of record of coverage, even if it is manually generated on local machines.

About

A simple Terraform module that sets log retention policies for all CloudWatch Log Groups so that you don't have to!

License:Apache License 2.0

Languages

Language:Rust 75.0%Language:HCL 23.5%Language:Shell 1.5%