A Terraform module to standardise S3 buckets with sensible defaults.
module "s3-bucket" {
source = "github.com/ministryofjustice/modernisation-platform-terraform-s3-bucket?ref=v7.0.0"
bucket_prefix = "s3-bucket"
versioning_enabled = true
# to disable ACLs in preference of BucketOwnership controls as per https://aws.amazon.com/blogs/aws/heads-up-amazon-s3-security-changes-are-coming-in-april-of-2023/ set:
ownership_controls = "BucketOwnerEnforced"
# Refer to the below section "Replication" before enabling replication
replication_enabled = false
# Below two variables and providers configuration are only relevant if 'replication_enabled' is set to true
# replication_region = "eu-west-2"
# replication_role_arn = module.s3-bucket-replication-role.role.arn
providers = {
# Here we use the default provider Region for replication. Destination buckets can be within the same Region as the
# source bucket. On the other hand, if you need to enable cross-region replication, please contact the Modernisation
# Platform team to add a new provider for the additional Region.
# Leave this provider block in even if you are not using replication
aws.bucket-replication = aws
}
lifecycle_rule = [
{
id = "main"
enabled = "Enabled"
prefix = ""
tags = {
rule = "log"
autoclean = "true"
}
transition = [
{
days = 90
storage_class = "STANDARD_IA"
}, {
days = 365
storage_class = "GLACIER"
}
]
expiration = {
days = 730
}
noncurrent_version_transition = [
{
days = 90
storage_class = "STANDARD_IA"
}, {
days = 365
storage_class = "GLACIER"
}
]
noncurrent_version_expiration = {
days = 730
}
}
]
tags = local.tags
}
| Name | Version |
|---|---|
| terraform | >= 1.0.1 |
| aws | ~> 5.0 |
| Name | Version |
|---|---|
| aws | ~> 5.0 |
| aws.bucket-replication | ~> 5.0 |
No modules.
| Name | Description | Type | Default | Required |
|---|---|---|---|---|
| acl | Use canned ACL on the bucket instead of BucketOwnerEnforced ownership controls. var.ownership_controls must be set to corresponding value below. | string |
"private" |
no |
| bucket_name | Please use bucket_prefix instead of bucket_name to ensure a globally unique name. | string |
null |
no |
| bucket_policy | JSON for the bucket policy | list(string) |
[ |
no |
| bucket_policy_v2 | Alternative to bucket_policy. Define policies directly without needing to know the bucket ARN | list(object({ |
[] |
no |
| bucket_prefix | Bucket prefix, which will include a randomised suffix to ensure globally unique names | string |
null |
no |
| custom_kms_key | KMS key ARN to use | string |
"" |
no |
| custom_replication_kms_key | KMS key ARN to use for replication to eu-west-2 | string |
"" |
no |
| force_destroy | A boolean that indicates all objects (including any locked objects) should be deleted from the bucket so that the bucket can be destroyed without error. These objects are not recoverable. | bool |
false |
no |
| lifecycle_rule | List of maps containing configuration of object lifecycle management. | any |
[ |
no |
| log_bucket | Bucket for server access logging, if applicable | string |
"" |
no |
| log_prefix | Prefix to use for server access logging, if applicable | string |
"" |
no |
| notification_enabled | Boolean indicating if a notification resource is required for the bucket | bool |
false |
no |
| notification_events | The event for which we send notifications | list(string) |
[ |
no |
| notification_sns_arn | The arn for the bucket notification SNS topic | string |
"" |
no |
| ownership_controls | Bucket Ownership Controls - for use WITH acl var above options are 'BucketOwnerPreferred' or 'ObjectWriter'. To disable ACLs and use new AWS recommended controls set this to 'BucketOwnerEnforced' and which will disabled ACLs and ignore var.acl | string |
"ObjectWriter" |
no |
| replication_enabled | Activate S3 bucket replication | bool |
false |
no |
| replication_region | Region to create S3 replication bucket | string |
"eu-west-2" |
no |
| replication_role_arn | Role ARN to access S3 and replicate objects | string |
"" |
no |
| sse_algorithm | The server-side encryption algorithm to use | string |
"aws:kms" |
no |
| tags | Tags to apply to resources, where applicable | map(any) |
n/a | yes |
| versioning_enabled | Activate S3 bucket versioning | bool |
true |
no |
| Name | Description |
|---|---|
| bucket | Direct aws_s3_bucket resource with all attributes |
| bucket_notifications | n/a |
| bucket_server_side_encryption | Bucket server-side encryption configuration |
Version 6.0.0 of this module uses the Hashicorp AWS Provider 4.0 as a minimum.
AWS Provider 4.0 introduces some significant changes to the s3_bucket resources as documented here.
We have worked to make the change as seamless to your code as possible, but you should expect to update your value for
Status from a boolean value of true | false to a string value of Enabled | Disabled.
Regardless of whether a custom bucket policy is set as part of this module, we will always include policy statement to require the use of SecureTransport (SSL) for every action on and every resource within the bucket.
If replication is enabled then:
- 'custom_replication_kms_key' variable is required, this key must allow access for S3
- 'versioning_enabled' variable must be set to enabled
- 'replication_role_arn' variable must be set to relevant arn for iam role
There are two ways to create the IAM role for replication:
- use the modernisation-platform-terraform-s3-bucket-replication-role to configure a role based on bucket ARNs
- create one yourself, by following the Setting up permissions for replication guide on AWS
See the aws_s3_bucket attributes reference. This module outputs the resource map, i.e. aws_s3_bucket, so you can access each attribute from Terraform directly under the bucket output, e.g. module.s3-bucket.bucket.id for the bucket ID.
If you're looking to raise an issue with this module, please create a new issue in the Modernisation Platform repository.
S3 is not suitable to store application logs directly but is ok for archived logs
- S3 is a bad idea for log files, since you cannot append to an object in S3. For every line in the log you'd have to download the file, append it and upload again, or make a new S3 object for every line in the log, which is highly inefficient. User data that doesn't change too often (like only a couple times a day or less) makes sense in S3. Something that changes all the time might make more sense in a database (stored on EBS).
- If you want to send logs directly to S3, you generate log files locally and save them to S3 periodically. For instance rotate your log files every minute and then send the old ones to S3.
Every version is charged as an individual object
- Normal Amazon S3 rates apply for every version of an object stored and transferred. Each version of an object is the entire object; it is not just a diff from the previous version. Thus, if you have three versions of an object stored, you are charged for three objects.
Versioning allows recovering files that are accidentally deleted
- With versioning you can recover more easily from both unintended user actions and application failures. Versioning-enabled buckets can help you recover objects from accidental deletion or overwrite. For example, if you delete an object, Amazon S3 inserts a delete marker instead of removing the object permanently. If you overwrite an object, it results in a new object version in the bucket. After you version-enable a bucket, it can never return to an unversioned state. But you can suspend versioning on that bucket.
Versioning requires separate lifecycle management configuration
- If you have versioning enabled, then in addition to the lifecycle policy for the current version you will need to configure a lifecycle policy for noncurrent versions. Otherwise, older versions will never be moved to cheaper storage and will never be expired/deleted.
References