Using "domain_name" in context with valid DNS completely breaks deployment

Question

Using "domain_name" in context with valid DNS completely breaks deployment

armenr opened this issue 3 years ago · comments

I have followed the steps in the Activate tutorial: https://activate.workshop.aws/

After bootstrapping and deploying the SDLC Org, I cut my DreamHost domain DNS over to the created Route53 zone in the "main" account of my new AWS setup.

I waited 8 hours for DNS to propagate, and checked it from multiple NS servers using dig

When I try to deploy my FrontendPipeline, deployment breaks at the Staging deployment step.

Note: If I run the deployment without "domain_name" context, deployment seems to work without issue., but that's not the desired outcome :)

I'm thinking about a few things:

Does this have to do with the permissionBoundary we set as an aspect of the PipelineStack?

// Respect cdk bootstrap policy insuring pipelines construct can't create more than what it needs for CI/CD pipeline creation
const permissionBoundaryArn = cdk.Fn.importValue('CICDPipelinePermissionsBoundaryArn');
cdk.Aspects.of(pipelineStack).add(new AddPermissionsBoundaryToRoles(permissionBoundaryArn));

Is there a Route53 "Permission Set" I should be creating and assigning in the "AWS Accounts" section of my SSO Organization? If so, what permissions should those be, which SSO Groups and/or accounts should they be assigned and applied to?
Are there missing IAM roles with read/list/write permissions for Route53 in my "main" account, which the Pipeline in the CICD account should be able to assume, but cannot?
It looks like the Staging account is trying to get permissions/assumeRole to make changes to my "main" account's DNS zones. Is that the case? If so, is that the intended behavior?
Are Route53 zones and DNS Delegation even being set up correctly?
In my root account DNS, I have the following hosted zones:

redacted-domain.me
dev.redacted-domain.me
prod.redacted-domain.me
staging.redacted-domain.me

Inside of Hosted Zone redacted-domain.me (in root account) I have a record for `staging.redacted-domain.me:

staging.redacted-domain.me | NS | Simple | - |
ns-555.awsdns-05.net
ns-1506.awsdns-60.org
ns-269.awsdns-33.com
ns-1775.awsdns-29.co.uk

Inside of Hosted Zone staging.redacted-domain.me (in root account) I have the following:

staging.redacted-domain.me | NS | Simple | - |
ns-269.awsdns-33.com.ns-1775.awsdns-29.co.uk.ns-555.awsdns-05.net.ns-1506.awsdns-60.org.


staging.redacted-domain.me | SOA | Simple | - |
ns-269.awsdns-33.com. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400

...but in my staging account, I only have 1 hosted zone, named appname-web.staging.redacted-domain.me and no other zones, with the following records. Should this just be a DNS entry, as opposed to a zone?

appname-web.staging.redacted-domain.me | SOA | Simple | - | 
ns-287.awsdns-35.com. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400

appname-web.staging.redacted-domain.me | NS | Simple | - | 
ns-287.awsdns-35.com.
ns-919.awsdns-50.net.
ns-1768.awsdns-29.co.uk
.ns-1117.awsdns-11.org.

_4837233fe50fb47655752c4dda901bec.appname-web.staging.redacted-domain.me | CNAME | Simple | - |
_703233e4799c310561b5ec55b015e433.nfyddsqlcy.acm-validations.aws.

Please see logs below -->

Logs:

Stack:
staging-WebAppStack-FrontendStackNestedStackFrontendStackNestedStackResource7A0E341B-OQN6W90BEOT8

Logical ID:
subzoneDelegationHostedZone61C4EBB4

Physical ID:
<REDACTED ZONE Z0699...>

Type:

AWS::Route53::HostedZone

Status:
DELETE_FAILED

The specified hosted zone contains non-required resource record sets and so cannot be deleted. (Service: Route53, Status Code: 400, Request ID: 58e24c4a-e9b3-4407-9cab-65e6c22cccbb, Extended Request ID: null)

The non-required resource in the Route53 zone file is the ACM validation entry that gets added while the stack is being deployed. From Route53.

Here's where it gets kinda bad. In my staging account, here's what I see in Route53:

Hosted Zones > appname-web.staging.redacted-domain.me

Records:

appname-web.staging.redacted-domain.me | NS | Simple --> 
ns-287.awsdns-35.com.
ns-919.awsdns-50.net.
ns-1768.awsdns-29.co.uk.
ns-1117.awsdns-11.org.

appname-web.staging.redacted-domain.me | SOA | Simple -->
ns-287.awsdns-35.com. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400

_4837233fe50fb47655752c4dda901bec.appname-web.staging.redacted-domain.me | CNAME | Simple | -->
_703233e4799c310561b5ec55b015e433.nfyddsqlcy.acm-validations.aws.

CloudWatch Logs

Log groups > /aws/lambda/staging-WebAppStack-Front-CrossAccountZoneDelegati-TXZJK1L5YARH > 2021/03/06/[$LATEST]7aa99371b8224dcb8d08c5821bd6ca40

START RequestId: 6dd5bd2b-0260-41ac-aa08-1407c502d896 Version: $LATEST

2021-03-06T09:52:43.540Z	6dd5bd2b-0260-41ac-aa08-1407c502d896	INFO	Event: 
{
    "RequestType": "Create",
    "ServiceToken": "arn:aws:lambda:us-west-2:STAGING_ACCOUNT_NUMBER:function:staging-WebAppStack-Front-CrossAccountZoneDelegati-11KEBTW3HUW55",
    "ResponseURL": "https://cloudformation-custom-resource-response-uswest2.s3-us-west-2.amazonaws.com/arn%3Aaws%3Acloudformation%3Aus-west-2%3ASTAGING_ACCOUNT_NUMBER%3Astack/staging-WebAppStack-FrontendStackNestedStackFrontendStackNestedStackResource7A0E341B-OQN6W90BEOT8/6dc38630-7e61-11eb-b0f7-02cff3de376d%7CsubzoneDelegationCrossAccountZoneDelegationRecordCrossAccountZoneDelegationRecordappnamewebstagingredacted-domainmeD89FF7A1%7Ce6fa7aba-fdfd-4b2a-869f-1e238a9d04d6?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20210306T095241Z&X-Amz-SignedHeaders=host&X-Amz-Expires=7200&X-Amz-Credential=AKIA54RCMT6SM5P7WMPE%2F20210306%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=009b44ab5b5d0d3a7dc87849a149a10984b7499b82e6bb6067956d8fda642843",
    "StackId": "arn:aws:cloudformation:us-west-2:STAGING_ACCOUNT_NUMBER:stack/staging-WebAppStack-FrontendStackNestedStackFrontendStackNestedStackResource7A0E341B-OQN6W90BEOT8/6dc38630-7e61-11eb-b0f7-02cff3de376d",
    "RequestId": "e6fa7aba-fdfd-4b2a-869f-1e238a9d04d6",
    "LogicalResourceId": "subzoneDelegationCrossAccountZoneDelegationRecordCrossAccountZoneDelegationRecordappnamewebstagingredacted-domainmeD89FF7A1",
    "ResourceType": "Custom::CrossAccountZoneDelegationRecord",
    "ResourceProperties": {
        "ServiceToken": "arn:aws:lambda:us-west-2:STAGING_ACCOUNT_NUMBER:function:staging-WebAppStack-Front-CrossAccountZoneDelegati-11KEBTW3HUW55",
        "currentAccountId": "STAGING_ACCOUNT_NUMBER",
        "toDelegateNameServers": [
            "ns-287.awsdns-35.com",
            "ns-919.awsdns-50.net",
            "ns-1768.awsdns-29.co.uk",
            "ns-1117.awsdns-11.org"
        ],
        "recordName": "appname-web.staging.redacted-domain.me"
    }
}

2021-03-06T09:52:45.029Z	6dd5bd2b-0260-41ac-aa08-1407c502d896	ERROR	Invoke Error 	
{
    "errorType": "AccessDenied",
    "errorMessage": "User: arn:aws:sts::STAGING_ACCOUNT_NUMBER:assumed-role/staging-WebAppStack-Front-CrossAccountZoneDelegati-QVR17XIY35S3/staging-WebAppStack-Front-CrossAccountZoneDelegati-TXZJK1L5YARH is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::MAIN_ACCOUNT_NUMBER:role/Staging.redacted-domain.me-dns-update",
    "code": "AccessDenied",
    "message": "User: arn:aws:sts::STAGING_ACCOUNT_NUMBER:assumed-role/staging-WebAppStack-Front-CrossAccountZoneDelegati-QVR17XIY35S3/staging-WebAppStack-Front-CrossAccountZoneDelegati-TXZJK1L5YARH is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::MAIN_ACCOUNT_NUMBER:role/Staging.redacted-domain.me-dns-update",
    "time": "2021-03-06T09:52:44.980Z",
    "requestId": "97877578-652d-46ca-a502-e051d704854e",
    "statusCode": 403,
    "retryable": false,
    "retryDelay": 76.96104528535399,
    "stack": [
        "AccessDenied: User: arn:aws:sts::STAGING_ACCOUNT_NUMBER:assumed-role/staging-WebAppStack-Front-CrossAccountZoneDelegati-QVR17XIY35S3/staging-WebAppStack-Front-CrossAccountZoneDelegati-TXZJK1L5YARH is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::MAIN_ACCOUNT_NUMBER:role/Staging.redacted-domain.me-dns-update",
        "    at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/query.js:50:29)",
        "    at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)",
        "    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)",
        "    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:688:14)",
        "    at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)",
        "    at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)",
        "    at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10",
        "    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)",
        "    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:690:12)",
        "    at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:116:18)"
    ]
}

END RequestId: 6dd5bd2b-0260-41ac-aa08-1407c502d896

Log groups > /aws/lambda/staging-WebAppStack-Front-CrossAccountZoneDelegati-11KEBTW3HUW55 > 2021/03/06/[$LATEST]94540ec5bbc44794a7b6ce5ac805f5a5

2021-03-06T09:52:42.014Z	b3eb75eb-d36f-4d16-bd9d-6827d73cb763	INFO	[provider-framework] onEventHandler 
{
    "RequestType": "Create",
    "ServiceToken": "arn:aws:lambda:us-west-2:STAGING_ACCOUNT_NUMBER:function:staging-WebAppStack-Front-CrossAccountZoneDelegati-11KEBTW3HUW55",
    "ResponseURL": "https://cloudformation-custom-resource-response-uswest2.s3-us-west-2.amazonaws.com/arn%3Aaws%3Acloudformation%3Aus-west-2%3ASTAGING_ACCOUNT_NUMBER%3Astack/staging-WebAppStack-FrontendStackNestedStackFrontendStackNestedStackResource7A0E341B-OQN6W90BEOT8/6dc38630-7e61-11eb-b0f7-02cff3de376d%7CsubzoneDelegationCrossAccountZoneDelegationRecordCrossAccountZoneDelegationRecordappnamewebstagingredacted-domainmeD89FF7A1%7Ce6fa7aba-fdfd-4b2a-869f-1e238a9d04d6?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20210306T095241Z&X-Amz-SignedHeaders=host&X-Amz-Expires=7200&X-Amz-Credential=AKIA54RCMT6SM5P7WMPE%2F20210306%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=009b44ab5b5d0d3a7dc87849a149a10984b7499b82e6bb6067956d8fda642843",
    "StackId": "arn:aws:cloudformation:us-west-2:STAGING_ACCOUNT_NUMBER:stack/staging-WebAppStack-FrontendStackNestedStackFrontendStackNestedStackResource7A0E341B-OQN6W90BEOT8/6dc38630-7e61-11eb-b0f7-02cff3de376d",
    "RequestId": "e6fa7aba-fdfd-4b2a-869f-1e238a9d04d6",
    "LogicalResourceId": "subzoneDelegationCrossAccountZoneDelegationRecordCrossAccountZoneDelegationRecordappnamewebstagingredacted-domainmeD89FF7A1",
    "ResourceType": "Custom::CrossAccountZoneDelegationRecord",
    "ResourceProperties": {
        "ServiceToken": "arn:aws:lambda:us-west-2:STAGING_ACCOUNT_NUMBER:function:staging-WebAppStack-Front-CrossAccountZoneDelegati-11KEBTW3HUW55",
        "currentAccountId": "STAGING_ACCOUNT_NUMBER",
        "toDelegateNameServers": [
            "ns-287.awsdns-35.com",
            "ns-919.awsdns-50.net",
            "ns-1768.awsdns-29.co.uk",
            "ns-1117.awsdns-11.org"
        ],
        "recordName": "appname-web.staging.redacted-domain.me"
    }
}

2021-03-06T09:52:42.036Z	b3eb75eb-d36f-4d16-bd9d-6827d73cb763	INFO	[provider-framework] executing user function arn:aws:lambda:us-west-2:STAGING_ACCOUNT_NUMBER:function:staging-WebAppStack-Front-CrossAccountZoneDelegati-TXZJK1L5YARH with payload 
{
    "RequestType": "Create",
    "ServiceToken": "arn:aws:lambda:us-west-2:STAGING_ACCOUNT_NUMBER:function:staging-WebAppStack-Front-CrossAccountZoneDelegati-11KEBTW3HUW55",
    "ResponseURL": "https://cloudformation-custom-resource-response-uswest2.s3-us-west-2.amazonaws.com/arn%3Aaws%3Acloudformation%3Aus-west-2%3ASTAGING_ACCOUNT_NUMBER%3Astack/staging-WebAppStack-FrontendStackNestedStackFrontendStackNestedStackResource7A0E341B-OQN6W90BEOT8/6dc38630-7e61-11eb-b0f7-02cff3de376d%7CsubzoneDelegationCrossAccountZoneDelegationRecordCrossAccountZoneDelegationRecordappnamewebstagingredacted-domainmeD89FF7A1%7Ce6fa7aba-fdfd-4b2a-869f-1e238a9d04d6?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20210306T095241Z&X-Amz-SignedHeaders=host&X-Amz-Expires=7200&X-Amz-Credential=AKIA54RCMT6SM5P7WMPE%2F20210306%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=009b44ab5b5d0d3a7dc87849a149a10984b7499b82e6bb6067956d8fda642843",
    "StackId": "arn:aws:cloudformation:us-west-2:STAGING_ACCOUNT_NUMBER:stack/staging-WebAppStack-FrontendStackNestedStackFrontendStackNestedStackResource7A0E341B-OQN6W90BEOT8/6dc38630-7e61-11eb-b0f7-02cff3de376d",
    "RequestId": "e6fa7aba-fdfd-4b2a-869f-1e238a9d04d6",
    "LogicalResourceId": "subzoneDelegationCrossAccountZoneDelegationRecordCrossAccountZoneDelegationRecordappnamewebstagingredacted-domainmeD89FF7A1",
    "ResourceType": "Custom::CrossAccountZoneDelegationRecord",
    "ResourceProperties": {
        "ServiceToken": "arn:aws:lambda:us-west-2:STAGING_ACCOUNT_NUMBER:function:staging-WebAppStack-Front-CrossAccountZoneDelegati-11KEBTW3HUW55",
        "currentAccountId": "STAGING_ACCOUNT_NUMBER",
        "toDelegateNameServers": [
            "ns-287.awsdns-35.com",
            "ns-919.awsdns-50.net",
            "ns-1768.awsdns-29.co.uk",
            "ns-1117.awsdns-11.org"
        ],
        "recordName": "appname-web.staging.redacted-domain.me"
    }
}

2021-03-06T09:52:45.078Z	b3eb75eb-d36f-4d16-bd9d-6827d73cb763	INFO	[provider-framework] user function response: 
{
    "StatusCode": 200,
    "FunctionError": "Unhandled",
    "ExecutedVersion": "$LATEST",
    "Payload": "{\"errorType\":\"AccessDenied\",\"errorMessage\":\"User: arn:aws:sts::STAGING_ACCOUNT_NUMBER:assumed-role/staging-WebAppStack-Front-CrossAccountZoneDelegati-QVR17XIY35S3/staging-WebAppStack-Front-CrossAccountZoneDelegati-TXZJK1L5YARH is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::MAIN_ACCOUNT_NUMBER:role/Staging.redacted-domain.me-dns-update\",\"trace\":[\"AccessDenied: User: arn:aws:sts::STAGING_ACCOUNT_NUMBER:assumed-role/staging-WebAppStack-Front-CrossAccountZoneDelegati-QVR17XIY35S3/staging-WebAppStack-Front-CrossAccountZoneDelegati-TXZJK1L5YARH is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::MAIN_ACCOUNT_NUMBER:role/Staging.redacted-domain.me-dns-update\",\"    at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/query.js:50:29)\",\"    at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)\",\"    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)\",\"    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:688:14)\",\"    at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)\",\"    at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)\",\"    at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10\",\"    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)\",\"    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:690:12)\",\"    at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:116:18)\"]}"
}
 object

 2021-03-06T09:52:45.116Z	b3eb75eb-d36f-4d16-bd9d-6827d73cb763	INFO	[provider-framework] user function threw an error: Unhandled

 2021-03-06T09:52:45.116Z	b3eb75eb-d36f-4d16-bd9d-6827d73cb763	INFO	[provider-framework] CREATE failed, responding with a marker physical resource id so that the subsequent DELETE will be ignored

 2021-03-06T09:52:45.116Z	b3eb75eb-d36f-4d16-bd9d-6827d73cb763	INFO	[provider-framework] submit response to cloudformation 
{
    "Status": "FAILED",
    "Reason": "User: arn:aws:sts::STAGING_ACCOUNT_NUMBER:assumed-role/staging-WebAppStack-Front-CrossAccountZoneDelegati-QVR17XIY35S3/staging-WebAppStack-Front-CrossAccountZoneDelegati-TXZJK1L5YARH is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::MAIN_ACCOUNT_NUMBER:role/Staging.redacted-domain.me-dns-update\n\nLogs: /aws/lambda/staging-WebAppStack-Front-CrossAccountZoneDelegati-TXZJK1L5YARH\n\n    at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/query.js:50:29)\n    at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)\n    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)\n    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:688:14)\n    at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)\n    at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)\n    at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10\n    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)\n    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:690:12)\n    at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:116:18)",
    "StackId": "arn:aws:cloudformation:us-west-2:STAGING_ACCOUNT_NUMBER:stack/staging-WebAppStack-FrontendStackNestedStackFrontendStackNestedStackResource7A0E341B-OQN6W90BEOT8/6dc38630-7e61-11eb-b0f7-02cff3de376d",
    "RequestId": "e6fa7aba-fdfd-4b2a-869f-1e238a9d04d6",
    "PhysicalResourceId": "AWSCDK::CustomResourceProviderFramework::CREATE_FAILED",
    "LogicalResourceId": "subzoneDelegationCrossAccountZoneDelegationRecordCrossAccountZoneDelegationRecordappnamewebstagingredacted-domainmeD89FF7A1"
}

END RequestId: b3eb75eb-d36f-4d16-bd9d-6827d73cb763

REPORT RequestId: b3eb75eb-d36f-4d16-bd9d-6827d73cb763	Duration: 3243.30 ms	Billed Duration: 3244 ms	Memory Size: 128 MB	Max Memory Used: 90 MB	Init Duration: 428.26 ms	

START RequestId: f39e7437-d870-4de7-b968-2cd745a099c0 Version: $LATEST

2021-03-06T09:53:35.249Z	f39e7437-d870-4de7-b968-2cd745a099c0	INFO	[provider-framework] ignoring DELETE event caused by a failed CREATE event

2021-03-06T09:53:35.249Z	f39e7437-d870-4de7-b968-2cd745a099c0	INFO	[provider-framework] submit response to cloudformation 
{
    "Status": "SUCCESS",
    "Reason": "SUCCESS",
    "StackId": "arn:aws:cloudformation:us-west-2:STAGING_ACCOUNT_NUMBER:stack/staging-WebAppStack-FrontendStackNestedStackFrontendStackNestedStackResource7A0E341B-OQN6W90BEOT8/6dc38630-7e61-11eb-b0f7-02cff3de376d",
    "RequestId": "cee9bfd0-1745-4558-abf4-5c75c7148fb2",
    "PhysicalResourceId": "AWSCDK::CustomResourceProviderFramework::CREATE_FAILED",
    "LogicalResourceId": "subzoneDelegationCrossAccountZoneDelegationRecordCrossAccountZoneDelegationRecordappnamewebstagingredacted-domainmeD89FF7A1"
}

END RequestId: f39e7437-d870-4de7-b968-2cd745a099c0

Sorry for the terrible formatting (I tried). Also, I'd just like to say thank you again for all of the incredible work you've all put into this project!

Any help or direction would be much appreciated. Thank you!

Armen Rostamian · Answer 1 · Sun Mar 07 2021 19:10:04 GMT+0800 (China Standard Time)

I spent some time exploring all of my Org's accounts and IAM policies, and I also read through all of the source code in your aws-bootstrap-kit npm package repo...both for the dns construct and the org construct that creates the IAM Roles and DNS Zones that show up in the stack (REALLY COOL STUFF BTW!).

Here's more useful info:

In "main" account, there is an IAM role for staging.redacted-domain.me-dns-update

1 Permissions policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "route53:GetHostedZone",
                "route53:ChangeResourceRecordSets",
                "route53:TestDNSAnswer"
            ],
            "Resource": "arn:aws:route53:::hostedzone/STAGING_ZONE_ID", (checked it, and it was correct)
            "Effect": "Allow"
        },
        {
            "Action": "route53:ListHostedZonesByName",
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

Trust relationship:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::STAGING_ACCOUNT_NUMBER:root"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

...So, at this point, I'm at a loss for understanding what could be causing this. The only thing that I can think of is that maybe the Lambda that assumes the dns-update role is being created/executed before the CrossAccountZoneDelegationRecordProviderframeworkonEvent IAM Role/Policy resources are created in the CloudFormation stack in the Staging account.

Here are the CloudFormation Stack Events:

CREATE_COMPLETE	- 2021-03-07 01:05:29 UTC-0800	CDKMetadata
CREATE_COMPLETE	- 2021-03-07 01:05:30 UTC-0800	SiteOAI1F7E9870
CREATE_COMPLETE	- 2021-03-07 01:05:34 UTC-0800	DeployWithInvalidationAwsCliLayerDEDD5787
CREATE_COMPLETE	- 2021-03-07 01:05:44 UTC-0800	CertificateCertificateRequestorFunctionServiceRoleC04C13DA
CREATE_COMPLETE	- 2021-03-07 01:05:44 UTC-0800	CrossAccountZoneDelegationRecordProviderOnEventHandlerServiceRoleA49D379C
CREATE_COMPLETE	- 2021-03-07 01:05:44 UTC-0800	CrossAccountZoneDelegationRecordProviderframeworkonEventServiceRoleA12ABBE7
CREATE_COMPLETE	- 2021-03-07 01:05:44 UTC-0800	CustomCDKBucketDeployment8693BB64968944B69AAFB0CC9EB8756CServiceRole89A01265
CREATE_COMPLETE	- 2021-03-07 01:05:50 UTC-0800	SiteBucket397A1860
CREATE_COMPLETE	- 2021-03-07 01:05:54 UTC-0800	SiteBucketPolicy3AC1D0F8
CREATE_COMPLETE	- 2021-03-07 01:06:02 UTC-0800	subzoneDelegationHostedZone61C4EBB4
CREATE_COMPLETE	- 2021-03-07 01:06:03 UTC-0800	CrossAccountZoneDelegationRecordProviderOnEventHandlerServiceRoleDefaultPolicy9E2936A7
CREATE_COMPLETE	- 2021-03-07 01:06:06 UTC-0800	CrossAccountZoneDelegationRecordProviderOnEventHandlerEB6D893E
CREATE_COMPLETE	- 2021-03-07 01:06:08 UTC-0800	CustomCDKBucketDeployment8693BB64968944B69AAFB0CC9EB8756CServiceRoleDefaultPolicy88902FDF
CREATE_COMPLETE	- 2021-03-07 01:06:14 UTC-0800	CustomCDKBucketDeployment8693BB64968944B69AAFB0CC9EB8756C81C01536
CREATE_COMPLETE	- 2021-03-07 01:06:21 UTC-0800	CertificateCertificateRequestorFunctionServiceRoleDefaultPolicy3C8845BC
CREATE_COMPLETE	- 2021-03-07 01:06:24 UTC-0800	CertificateCertificateRequestorFunction5E845413
CREATE_COMPLETE	- 2021-03-07 01:06:24 UTC-0800	CrossAccountZoneDelegationRecordProviderframeworkonEventServiceRoleDefaultPolicy0D48FB87
CREATE_COMPLETE	- 2021-03-07 01:06:27 UTC-0800	CrossAccountZoneDelegationRecordProviderframeworkonEventC430360E

Everything looks like it was created in the right sequence/order, right? The only thing I can think of is that the CrossAccountZoneDelegationRecordProviderOnEventHandlerServiceRole and CrossAccountZoneDelegationRecordProviderframeworkonEventServiceRole should have a dependsOn to the CrossAccountZoneDelegationRecordProviderframeworkonEventServiceRoleDefaultPolicy and CrossAccountZoneDelegationRecordProviderOnEventHandlerServiceRoleDefaultPolicy so that you have policies created BEFORE the service roles get created...but that's just a theory.

...I seriously can't figure out why the Lambda isn't able to assume the role in the main account. Any help would VERY much appreciated. Thank you again!

Armen Rostamian · Answer 2 · Tue Mar 09 2021 08:04:15 GMT+0800 (China Standard Time)

Confirmed working/resolved. Thank you! You're the man, @flochaz !