Two-way replication configuration does not complete

Question

Two-way replication configuration does not complete

sbrinkerhoff opened this issue 2 years ago · comments

Description

I am attempting to configure two-way S3 bucket replication, and receive an error that InvalidRequest: Destination bucket must have versioning enabled. when doing so. Immediately issuing a subsequent terraform apply completes with no errors and a successful configuration.

Alternatively creating the buckets and then adding the replication configuration in afterwards works fine with no intermediate error.

This is similar to #42, except the issue references in #42 was resolved in upstream.

✋ I have searched the open/closed issues and my issue is not listed.

Versions

Module version [Required]:

v3.3.0

Terraform version:
Terraform 1.1.9
Provider version(s):
Terraform v1.1.9
on darwin_amd64

provider registry.terraform.io/gavinbunney/kubectl v1.14.0
provider registry.terraform.io/hashicorp/aws v4.24.0
provider registry.terraform.io/hashicorp/cloudinit v2.2.0
provider registry.terraform.io/hashicorp/helm v2.6.0
provider registry.terraform.io/hashicorp/kubernetes v2.10.0
provider registry.terraform.io/hashicorp/random v3.3.2
provider registry.terraform.io/hashicorp/time v0.7.2
provider registry.terraform.io/hashicorp/tls v3.4.0

Reproduction Code [Required]

locals {
  asset_primary_name   = "assets-${local.environment}-${local.regions.primary}-${random_pet.asset_random.id}"
  asset_secondary_name = "assets-${local.environment}-${local.regions.secondary}-${random_pet.asset_random.id}"
}

resource "random_pet" "asset_random" {
}

resource "aws_iam_role_policy" "s3_policy" {
  name = "asset-${local.environment}-policy"
  role = var.arn
  policy = data.aws_iam_policy_document.document.json
}

data "aws_iam_policy_document" "document" {
  statement {
    sid    = "1"
    effect = "Allow"
    actions = [
      "s3:GetObjectVersionForReplication",
      "s3:GetObjectVersionAcl",
      "s3:GetObjectVersionTagging",
      "s3:ReplicateObject",
      "s3:ReplicateDelete",
      "s3:ReplicateTags",
      "s3:GetReplicationConfiguration",
      "s3:ListBucket"
    ]

    resources = [
      "${module.assets_primary.s3_bucket_arn}/*",
      "${module.assets_secondary.s3_bucket_arn}/*"
    ]
  }
}

module "assets_primary" {
  source  = "terraform-aws-modules/s3-bucket/aws"
  version = "3.3.0"

  bucket = local.asset_primary_name
  acl    = "private"

  versioning = {
    enabled = true
  }

  # S3 bucket-level Public Access Block configuration
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true

  # https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_ownership_controls
  control_object_ownership = true
  object_ownership         = "BucketOwnerPreferred"

  server_side_encryption_configuration = {
    rule = {
      apply_server_side_encryption_by_default = {
        kms_master_key_id = data.aws_kms_key.key.arn
        sse_algorithm     = "aws:kms"
      }
    }
  }

  replication_configuration = {
    role = var.arn
    rules = {
      id                        = "replicate-all-objects"
      status                    = true
      priority                  = "10"
      delete_marker_replication = true
      source_selection_criteria = {
        replica_modifications = {
          status = "Enabled"
        }
        sse_kms_encrypted_objects = {
          enabled = true
        }
      }
      destination = {
        bucket     = "arn:aws:s3:::${local.asset_secondary_name}"
        account_id = data.aws_caller_identity.current.account_id

        replica_kms_key_id = data.aws_kms_key.key-us-west-2.arn
        access_control_translation = {
          owner = "Destination"
        }

        replication_time = {
          status  = "Enabled"
          minutes = 15
        }

        metrics = {
          status  = "Enabled"
          minutes = 15
        }
      }
    }
  }
}

module "assets_secondary" {
  source  = "terraform-aws-modules/s3-bucket/aws"
  version = "3.3.0"
  providers = {
    aws = aws.us-west-2
  }

  bucket = local.asset_secondary_name
  acl    = "private"

  versioning = {
    enabled = true
  }

  # S3 bucket-level Public Access Block configuration
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true

  # https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_ownership_controls
  control_object_ownership = true
  object_ownership         = "BucketOwnerPreferred"

  server_side_encryption_configuration = {
    rule = {
      apply_server_side_encryption_by_default = {
        kms_master_key_id = data.key.arn
        sse_algorithm     = "aws:kms"
      }
    }
  }

  replication_configuration = {
    role = var.arn
    rules = {
      id                        = "replicate-all-objects"
      status                    = true
      priority                  = "10"
      delete_marker_replication = true
      source_selection_criteria = {
        replica_modifications = {
          status = "Enabled"
        }
        sse_kms_encrypted_objects = {
          enabled = true
        }
      }
      destination = {
        bucket     = "arn:aws:s3:::${local.asset_primary_name}"
        account_id = data.aws_caller_identity.current.account_id

        replica_kms_key_id = data.aws_kms_key.key.arn
        access_control_translation = {
          owner = "Destination"
        }

        replication_time = {
          status  = "Enabled"
          minutes = 15
        }

        metrics = {
          status  = "Enabled"
          minutes = 15
        }
      }
    }
  }
}

Steps to reproduce the behavior:

Are you using workspaces?

Yes.

Have you cleared the local cache (see Notice section above)?

Yes.

List steps in order that led up to the issue you encountered

Terraform apply.

Expected behavior

Two buckets with replication would be created with a single terraform apply without error.

Actual behavior

│ Error: error creating S3 replication configuration for bucket (assets-dev-us-east-1-enjoyed-koala): InvalidRequest: Destination bucket must have versioning enabled.
│ 	status code: 400, request id: 3SWPGQ12GXV1TS6S, host id: 0L6HAwm2PqjnL6jvM7dTX42cJvzfcUePCRWoDn9jw1RAqSyeUTbMuMvOAFPS18gL5Jr7MoS9qak=
│
│   with module.assets_primary.aws_s3_bucket_replication_configuration.this[0],
│   on .terraform/modules/assets_primary/main.tf line 354, in resource "aws_s3_bucket_replication_configuration" "this":
│  354: resource "aws_s3_bucket_replication_configuration" "this" {
│

Terminal Output Screenshot(s)

See above.

Additional Detail

This appears to be a race condition of sorts --

If I create the two buckets, and then add the replication configuration in, it adds just fine.
If I run terraform apply, let it fail, and then terraform apply again it works just fine.
Checking the bucket immediately after the first fail, the secondary bucket does have versioning enabled.

samsolaimani · Answer 1 · Tue Aug 16 2022 22:37:13 GMT+0800 (China Standard Time)

Is there any update on this, GitHub Support?

Anton Babenko · Answer 2 · Wed Aug 17 2022 20:34:57 GMT+0800 (China Standard Time)

I took a look into the provided code and I could not reproduce this issue using the provided configuration.

Here is the configuration I used - all in one file, based on your code. It creates resources as expected but I didn't check whether it configures a two-way replication correctly.

locals {
  environment = "yo"
  regions = {
    primary = "eu-west-1"
    secondary = "eu-central-1"
  }
  asset_primary_name   = "assets-${local.environment}-${local.regions.primary}-${random_pet.asset_random.id}"
  asset_secondary_name = "assets-${local.environment}-${local.regions.secondary}-${random_pet.asset_random.id}"
}


provider "aws" {
  region = local.regions.primary

  # Make it faster by skipping something
  skip_get_ec2_platforms      = true
  skip_metadata_api_check     = true
  skip_region_validation      = true
  skip_credentials_validation = true
  skip_requesting_account_id  = true
}

provider "aws" {
  region = local.regions.secondary

  alias = "replica"

  # Make it faster by skipping something
  skip_get_ec2_platforms      = true
  skip_metadata_api_check     = true
  skip_region_validation      = true
  skip_credentials_validation = true
  skip_requesting_account_id  = true
}

resource "random_pet" "asset_random" {
}

resource "aws_iam_role" "replication" {
  name = "s3-bucket-replication-${random_pet.asset_random.id}"

  assume_role_policy = <<POLICY
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "s3.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
POLICY
}

resource "aws_iam_role_policy" "s3_policy" {
  name = "asset-${local.environment}-policy"
  role = aws_iam_role.replication.id
  policy = data.aws_iam_policy_document.document.json
}

data "aws_iam_policy_document" "document" {
  statement {
    sid    = "1"
    effect = "Allow"
    actions = [
      "s3:GetObjectVersionForReplication",
      "s3:GetObjectVersionAcl",
      "s3:GetObjectVersionTagging",
      "s3:ReplicateObject",
      "s3:ReplicateDelete",
      "s3:ReplicateTags",
      "s3:GetReplicationConfiguration",
      "s3:ListBucket"
    ]

    resources = [
      "${module.assets_primary.s3_bucket_arn}/*",
      "${module.assets_secondary.s3_bucket_arn}/*"
    ]
  }
}

data "aws_caller_identity" "current" {}

resource "aws_kms_key" "primary" {
  description             = "S3 bucket KMS key"
  deletion_window_in_days = 7
}

resource "aws_kms_key" "replica" {
  provider = aws.replica

  description             = "S3 bucket replication KMS key"
  deletion_window_in_days = 7
}

module "assets_primary" {
  source  = "terraform-aws-modules/s3-bucket/aws"

  bucket = local.asset_primary_name
  acl    = "private"

  versioning = {
    enabled = true
  }

  # S3 bucket-level Public Access Block configuration
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true

  # https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_ownership_controls
  control_object_ownership = true
  object_ownership         = "BucketOwnerPreferred"

  server_side_encryption_configuration = {
    rule = {
      apply_server_side_encryption_by_default = {
        kms_master_key_id = aws_kms_key.primary.arn
        sse_algorithm     = "aws:kms"
      }
    }
  }

  replication_configuration = {
    role = aws_iam_role.replication.arn
    rules = {
      id                        = "replicate-all-objects"
      status                    = true
      priority                  = "10"
      delete_marker_replication = true
      source_selection_criteria = {
        replica_modifications = {
          status = "Enabled"
        }
        sse_kms_encrypted_objects = {
          enabled = true
        }
      }
      destination = {
        bucket     = "arn:aws:s3:::${local.asset_secondary_name}"
        account_id = data.aws_caller_identity.current.account_id

        replica_kms_key_id = aws_kms_key.replica.arn
        access_control_translation = {
          owner = "Destination"
        }

        replication_time = {
          status  = "Enabled"
          minutes = 15
        }

        metrics = {
          status  = "Enabled"
          minutes = 15
        }
      }
    }
  }
}

module "assets_secondary" {
  source  = "terraform-aws-modules/s3-bucket/aws"

  providers = {
    aws = aws.replica
  }

  bucket = local.asset_secondary_name
  acl    = "private"

  versioning = {
    enabled = true
  }

  # S3 bucket-level Public Access Block configuration
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true

  # https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_ownership_controls
  control_object_ownership = true
  object_ownership         = "BucketOwnerPreferred"

  server_side_encryption_configuration = {
    rule = {
      apply_server_side_encryption_by_default = {
        kms_master_key_id = aws_kms_key.replica.arn
        sse_algorithm     = "aws:kms"
      }
    }
  }

  replication_configuration = {
    role = aws_iam_role.replication.arn
    rules = {
      id                        = "replicate-all-objects"
      status                    = true
      priority                  = "10"
      delete_marker_replication = true
      source_selection_criteria = {
        replica_modifications = {
          status = "Enabled"
        }
        sse_kms_encrypted_objects = {
          enabled = true
        }
      }
      destination = {
        bucket     = "arn:aws:s3:::${local.asset_primary_name}"
        account_id = data.aws_caller_identity.current.account_id

        replica_kms_key_id = aws_kms_key.primary.arn
        access_control_translation = {
          owner = "Destination"
        }

        replication_time = {
          status  = "Enabled"
          minutes = 15
        }

        metrics = {
          status  = "Enabled"
          minutes = 15
        }
      }
    }
  }
}

github-actions · Answer 3 · Sat Sep 17 2022 08:03:35 GMT+0800 (China Standard Time)

This issue has been automatically marked as stale because it has been open 30 days
with no activity. Remove stale label or comment or this issue will be closed in 10 days

github-actions · Answer 4 · Tue Sep 27 2022 08:08:06 GMT+0800 (China Standard Time)

This issue was automatically closed because of stale in 10 days

github-actions · Answer 5 · Fri Oct 28 2022 10:40:28 GMT+0800 (China Standard Time)

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.