[Bug]: Org Trail resource creation failed for CloudTrail Delegated Administrator

Question

[Bug]: Org Trail resource creation failed for CloudTrail Delegated Administrator

mhgeay opened this issue a year ago · comments

Marc-Henry GEAY commented a year ago

Terraform Core Version

1.3.6

AWS Provider Version

4.47.0

Affected Resource(s)

aws_cloudtrail

Expected Behavior

We choose a Delegated Administrator for the CloudTrail Service (see AWS doc). All CloudTrail resources remain in the Management Account, even if they are created by the Delegated Account. (the ARN show the Account ID of the Management Account).
We deploy a new organization trail on this Delegated Account with Terraform.
The organization trail is fully created and enabled in the Management Account.

Actual Behavior

When Terraform applies, it's failed when it try to enable logging it in the Delegated Account because it's did not find the trail.

After successfully creation, The AWS provider seems try to enable the trail locally (in the Delegated account) instead to referer to the trail 'stored' in the Management Account. The ARN in the below log explains the bug.

Relevant Error/Panic Output Snippet

'111111111111' is the Delegated Account ID.

aws_cloudtrail.org_main: Creating...
╷
│ Error: Error starting logging on CloudTrail (org-cloudtrail): TrailNotFoundException: Unknown trail: arn:aws:cloudtrail:us-east-1:111111111111:trail/org-cloudtrail for the user: 111111111111
│ 
│   with aws_cloudtrail.org_main,
│   on 01-cloudtrail.tf line 4, in resource "aws_cloudtrail" "org_main":
│    4: resource "aws_cloudtrail" "org_main" {
│ 
╵
Operation failed: failed running terraform apply (exit 1)�

Terraform Configuration Files

resource "aws_cloudtrail" "org_main" {
  name                          = "org-cloudtrail"
  enable_logging                = true
  s3_bucket_name                = aws_s3_bucket.cloudtrail.id
  s3_key_prefix                 = ""
  include_global_service_events = true
  enable_log_file_validation    = false
  is_multi_region_trail         = true
  is_organization_trail         = true

  event_selector {
    read_write_type                  = "WriteOnly"
    include_management_events        = true
    exclude_management_event_sources = ["kms.amazonaws.com", "rdsdata.amazonaws.com"]
  }
}

Steps to Reproduce

TF plan (ok)
TF apply (failed)

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-delegated-administrator.html

Would you like to implement a fix?

No

Taylor Williams commented 7 months ago

Upvoting

github-actions · Answer 1 · Sun Dec 18 2022 22:55:55 GMT+0800 (China Standard Time)

Community Note

Voting for Prioritization

Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
Please see our prioritization guide for information on how we prioritize.
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

If you are interested in working on this issue, please leave a comment.
If this would be your first contribution, please review the contribution guide.

feng xu · Answer 2 · Mon Dec 19 2022 22:01:19 GMT+0800 (China Standard Time)

I have got the same issue.
The following error occur when "terraform apply" is run again:
"Error creating CloudTrail: TrailAlreadyExistsException: Trail [trailname] already exists for customer: 111111111 (Service: AWSCloudTrail; Status Code: 400; Error Code: TrailAlreadyExistsException;...".

Dave O · Answer 3 · Fri Jan 06 2023 15:22:08 GMT+0800 (China Standard Time)

Have the exact same issue occurring as the original poster, when i go into the AWS console i can see the trail has been created but has not been activated. If i click start logging it starts up fine, so could be the issue be a timing related one.

Also if i run the script a second i get the exact same output as F-Xu above, Terraform doesn't think it has already created the trail so its attempting to create it again.

Andrew Nicolalde · Answer 4 · Wed Jan 18 2023 03:22:52 GMT+0800 (China Standard Time)

I think I might have found a solution (more of a workaround really) to this issue in the interim, at least until the AWS Terraform provider can be updated to properly manage resources created from delegated administrator accounts. This was inspired by a bit of AWS sample code for using delegated administration with GuardDuty.

You can define a separate terraform provider for your management account

provider "aws" {
  region  = "us-east-1"
  profile = "security-acct"
}

provider "aws" {
  alias   = "mgmt"
  profile = "management-acct"
  region  = "us-east-1"
}

and use it to directly manage just the aws_cloudtrail resource.

resource "aws_cloudtrail" "org_trail" {
  provider = aws.mgmt
  name  = "org_trail"
  ...
}

I believe there is an underlying issue with the way AWS has implemented the GetTrail action that prevents the AWS terraform provider from implementing this properly though. It seems that Cloudtrail delegated administrator accounts don't have the ability to perform GetTrail against trails they created as delegated administrators, which are owned by the management account. This is especially weird given what the AWS docs say about the Cloudtrail delegated administration feature:

If you choose a delegated administrator, this member account has administrative permissions on all trails and event data stores in the organization. Adding a delegated administrator does not disrupt the management or operation of the organization's trails or event data stores.

If you run aws --profile security-acct --region us-east-1 cloudtrail describe-trails, you get

{
    "trailList": [
        {
            "Name": "regular-trail-1",
            ...
            "TrailARN": "arn:aws:cloudtrail:us-east-1:<SECURITY ACCOUNT ID>:trail/regular-trail-1",
            ...
        },
        {
            "Name": "regular-trail-2",
            ...
            "TrailARN": "arn:aws:cloudtrail:us-east-1:<SECURITY ACCOUNT ID>:trail/regular-trail-2",
            ...
        },
        {
            "Name": "org_trail",
            ...
            "TrailARN": "arn:aws:cloudtrail:us-east-1:<MANAGEMENT ACCOUNT ID>:trail/org_trail",
            ...
        }
    ]
}

but when you run aws --profile security-acct --region us-east-1 cloudtrail get-trail --name org_trail, you get back

An error occurred (TrailNotFoundException) when calling the GetTrail operation: Unknown trail: org_trail for the user: <SECURITY ACCOUNT ID>

When you try to run the same command but from your management account (i.e. aws --profile management-acct --region us-east-1 cloudtrail get-trail --name org_trail) , it works as expected.

Marc Zampetti · Answer 5 · Fri Mar 10 2023 02:47:39 GMT+0800 (China Standard Time)

As a follow up to @andrewnicolalde, if you run aws --profile security-acct --region us-east-1 cloudtrail get-trail --name <arn of org trail> instead of the trail name in the Delegated Admin account, you DO get the details back.

So it would seem a simple fix would be to update the terraform aws provider to use the ARN of the trail instead of the name. The other option would be to use the describe-trails without any option, which DOES return the org trail, and then iterate over the values find the one that matches the Name that is being requested. Not as efficient, but given that the vast majority of users are not going to have more than a small number of trails defined (there isn't much of a reasons to do so) it shouldn't be an issue.

Andrii Areshchenko · Answer 6 · Tue Jun 13 2023 00:50:41 GMT+0800 (China Standard Time)

Could you merge the PR please? Have the same issue

Serhii Khylyk · Answer 7 · Tue Jun 13 2023 00:51:27 GMT+0800 (China Standard Time)

and we too, so just waiting (

Anastazja · Answer 8 · Fri Jul 28 2023 21:50:45 GMT+0800 (China Standard Time)

I tried the workaround suggested by @andrewnicolalde and it didn't work. Could you please merge the PR- would make it amazing if this got fixed. 🥺

henning · Answer 9 · Mon Aug 07 2023 20:54:49 GMT+0800 (China Standard Time)

Having the same issue. Upvoting for fix/PR

Mummycutie · Answer 10 · Thu Aug 10 2023 15:26:33 GMT+0800 (China Standard Time)

Having same issue. Upvoting

pstsef-blackberry · Answer 11 · Fri Aug 11 2023 07:11:18 GMT+0800 (China Standard Time)

Having the same issue. Upvoting.

hedy kim · Answer 12 · Fri Aug 18 2023 20:17:15 GMT+0800 (China Standard Time)

Upvoting. 🔥

sidekick-eimantas · Answer 13 · Tue Aug 22 2023 00:31:55 GMT+0800 (China Standard Time)

@bflad @ewbankkit This is impacting quite a few people and is critical functionality for cloudtrail management
Could someone have a look at the PR above please?

Thank you

Paweł Słowik · Answer 14 · Thu Nov 02 2023 22:02:54 GMT+0800 (China Standard Time)

I have the same issue. Upvoting 👍🏻

github-actions · Answer 15 · Fri Nov 10 2023 15:15:56 GMT+0800 (China Standard Time)

This functionality has been released in v5.25.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

github-actions · Answer 16 · Mon Dec 11 2023 10:08:43 GMT+0800 (China Standard Time)

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.