Cannot assume roles with any version since 6.3.1

Question

Cannot assume roles with any version since 6.3.1

bryanhorstmann opened this issue 2 years ago · comments

I am using the latest release of AWS Vault
I have provided my .aws/config (redacted if necessary)
I have provided the debug output using aws-vault --debug (redacted if necessary)

I'm pulling my hair out trying to debug this. aws-vault works perfectly fine on 6.3.1. But if I update to any version afterward, I get the following error:

aws-vault: error: exec: Failed to get credentials for child: operation error STS: AssumeRole, https response error StatusCode: 403, RequestID: 59d8c36c-bfe9-467a-ae51-e01de21e8c73, api error AccessDenied: User: arn:aws:iam::111111111111:user/bryan.h is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::222222222222:role/OrganizationAccountAccessRole-Admin

I have tested with 6.4.0, 6.5.0, 6.6.0.

This issue is occuring for multiple users across the business. The only thing we all have in common is we're running Apple Silicon chipset.

My .aws/config

→ cat .aws/config
[profile parent]
mfa_serial=arn:aws:iam::111111111111:mfa/bryan.h
region=us-east-1

[profile child]
mfa_serial=arn:aws:iam::111111111111:mfa/bryan.h
source_profile=parent
region=eu-west-1
role_arn=arn:aws:iam::222222222222:role/OrganizationAccountAccessRole-Admin

Testing

Versions

→ aws-vault6.3.1 --version
v6.3.1

→ aws-vault6.6.0 --version
v6.6.0

Not assuming roles

→ aws-vault6.3.1 exec parent -- aws s3 ls
2022-06-28 09:55:20 bucket1
2022-06-27 15:53:09 bucket2
2020-06-30 11:01:59 bucket3
2019-11-25 13:38:34 bucket4

→ aws-vault6.6.0 exec parent -- aws s3 ls
2022-06-28 09:55:20 bucket1
2022-06-27 15:53:09 bucket2
2020-06-30 11:01:59 bucket3
2019-11-25 13:38:34 bucket4

Assuming roles

→ aws-vault6.3.1 exec child -- aws s3 ls
2021-10-29 09:17:23 bucket1
2020-06-30 10:59:22 bucket2
2020-05-07 10:25:33 bucket3
2021-04-15 11:20:03 bucket4

→ aws-vault6.6.0 exec child -- aws s3 ls
aws-vault: error: exec: Failed to get credentials for child: operation error STS: AssumeRole, https response error StatusCode: 403, RequestID: fb49ba0d-f9b7-40a3-b76c-d167603a7b7a, api error AccessDenied: User: arn:aws:iam::111111111111:user/bryan.h is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::222222222222:role/OrganizationAccountAccessRole-Admin

Debug logs

→ aws-vault6.6.0 exec yoco-aws-dev --debug -- aws s3 ls
2022/07/15 11:04:47 aws-vault v6.6.0
2022/07/15 11:04:47 Loading config file /Users/bryanhorstmann/.aws/config
2022/07/15 11:04:47 Parsing config file /Users/bryanhorstmann/.aws/config
2022/07/15 11:04:47 [keyring] Considering backends: [keychain]
2022/07/15 11:04:47 Profile 'default' missing in config file
2022/07/15 11:04:47 Profile 'default' missing in config file
2022/07/15 11:04:47 [keyring] Querying keychain for service="aws-vault", keychain="aws-vault.keychain"
2022/07/15 11:04:47 [keyring] Found 3 results
2022/07/15 11:04:47 [keyring] Querying keychain for service="aws-vault", keychain="aws-vault.keychain"
2022/07/15 11:04:47 [keyring] Found 3 results
2022/07/15 11:04:47 profile parent: using stored credentials
2022/07/15 11:04:47 profile parent: using GetSessionToken (with MFA)
2022/07/15 11:04:47 profile child: using AssumeRole (chained MFA)
2022/07/15 11:04:47 [keyring] Querying keychain for service="aws-vault", keychain="aws-vault.keychain"
2022/07/15 11:04:47 [keyring] Found 3 results
2022/07/15 11:04:47 [keyring] Querying keychain for service="aws-vault", keychain="aws-vault.keychain"
2022/07/15 11:04:47 [keyring] Found 3 results
2022/07/15 11:04:47 [keyring] Querying keychain for service="aws-vault", account="sts.GetSessionToken,eW9jby1hd3MtbWFzdGVy,YXJuOmF3czppYW06OjA1MjI5NDI1NTQwMjptZmEvYnJ5YW4uaA,1657904023", keychain="aws-vault.keychain"
2022/07/15 11:04:47 [keyring] Found item "aws-vault session for parent (expires 2022-07-15T16:53:43Z)"
2022/07/15 11:04:47 Re-using cached credentials ****************I2DX from sts.GetSessionToken, expires in 7h48m55.437109s
aws-vault: error: exec: Failed to get credentials for child: operation error STS: AssumeRole, https response error StatusCode: 403, RequestID: c44e0fb9-8586-495d-ba6d-8e09eddf6325, api error AccessDenied: User: arn:aws:iam::111111111111:user/bryan.h is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::222222222222:role/OrganizationAccountAccessRole-Admin

Yoriyasu Yano · Answer 1 · Wed Jul 20 2022 03:20:11 GMT+0800 (China Standard Time)

I had the same issue after upgrading, and was able to resolve this after adding a [default] profile with a region:

[default]
region = us-east-1

Bryan Horstmann · Answer 2 · Wed Aug 24 2022 17:18:32 GMT+0800 (China Standard Time)

Hi @yorinasub17,

I have added that and still have the same problem.

Michael Tibben · Answer 3 · Thu Sep 01 2022 06:31:02 GMT+0800 (China Standard Time)

The major change between 6.3.1 and 6.40 was the use of the v2 AWS SDK.

In your debug logs I note 2022/07/15 11:04:47 Re-using cached credentials, which means the v2 SDK is trying to re-use credentials generated by v1.

I would suggest firstly doing a aws-vault clear - you may find v6.4.0 works fine after doing this. If not, post the debug logs after doing the clear.

Secondly I would suggest trying this with aws-vault --debug exec --no-session

Bryan Horstmann · Answer 4 · Fri Sep 02 2022 20:26:50 GMT+0800 (China Standard Time)

Thanks for the feedback @mtibben. Still having issues unfortunately.

Order of events:

aws-vault clear
aws-vault6.3.1 exec child -- aws s3 ls works
aws-vault clear just to confirm its clear
aws-vault6.6.0 --debug exec child --no-session -- aws s3 ls fails

→ /usr/local/bin/aws-vault6.6.0 --debug exec child --no-session -- aws s3 ls
2022/09/02 14:18:04 aws-vault v6.6.0
2022/09/02 14:18:04 Loading config file /Users/bryanhorstmann/.aws/config
2022/09/02 14:18:04 Parsing config file /Users/bryanhorstmann/.aws/config
2022/09/02 14:18:04 [keyring] Considering backends: [keychain]
2022/09/02 14:18:04 [keyring] Querying keychain for service="aws-vault", keychain="aws-vault.keychain"
2022/09/02 14:18:04 [keyring] Found 3 results
2022/09/02 14:18:04 [keyring] Querying keychain for service="aws-vault", keychain="aws-vault.keychain"
2022/09/02 14:18:04 [keyring] Found 3 results
2022/09/02 14:18:04 profile parent: using stored credentials
2022/09/02 14:18:04 profile parent: skipping GetSessionToken because disabled
2022/09/02 14:18:04 profile child: using AssumeRole (with MFA)
2022/09/02 14:18:04 [keyring] Querying keychain for service="aws-vault", keychain="aws-vault.keychain"
2022/09/02 14:18:04 [keyring] Found 3 results
2022/09/02 14:18:04 [keyring] Querying keychain for service="aws-vault", keychain="aws-vault.keychain"
2022/09/02 14:18:04 [keyring] Found 3 results
2022/09/02 14:18:04 [keyring] Querying keychain for service="aws-vault", account="sts.AssumeRole,eW9jby1hd3MtZGV2,YXJuOmF3czppYW06OjA1MjI5NDI1NTQwMjptZmEvYnJ5YW4uaA,-62135596800", keychain="aws-vault.keychain"
2022/09/02 14:18:04 [keyring] No results found
Enter MFA code for arn:aws:iam::111111111111:mfa/bryan.h: 220360
2022/09/02 14:18:13 Looking up keyring for 'parent'
2022/09/02 14:18:13 [keyring] Querying keychain for service="aws-vault", account="parent", keychain="aws-vault.keychain"
2022/09/02 14:18:13 [keyring] Found item "aws-vault (parent)"
aws-vault: error: exec: Failed to get credentials for child: operation error STS: AssumeRole, https response error StatusCode: 403, RequestID: 7780ab14-72d9-4746-99f4-b1cfac6e82e5, api error AccessDenied: User: arn:aws:iam::111111111111:user/bryan.h is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::222222222222:role/OrganizationAccountAccessRole-Admin

I even tried, running the clear command against all the binaries I have on my machine and tried again. Same result.

# bryanhorstmann at THVQ46M6RN in ~ [14:22:38]
→ aws-vault clear
Cleared 1 sessions.

# bryanhorstmann at THVQ46M6RN in ~ [14:23:35]
→ aws-vault6.3.1 clear
Cleared 0 sessions.

# bryanhorstmann at THVQ46M6RN in ~ [14:23:40]
→ aws-vault6.6.0 clear
Cleared 0 sessions.

# bryanhorstmann at THVQ46M6RN in ~ [14:23:49]
→ aws-vault6.6.0 exec child --debug --no-session -- aws s3 ls
2022/09/02 14:24:20 aws-vault v6.6.0
2022/09/02 14:24:20 Loading config file /Users/bryanhorstmann/.aws/config
2022/09/02 14:24:20 Parsing config file /Users/bryanhorstmann/.aws/config
2022/09/02 14:24:20 [keyring] Considering backends: [keychain]
2022/09/02 14:24:20 [keyring] Querying keychain for service="aws-vault", keychain="aws-vault.keychain"
2022/09/02 14:24:20 [keyring] Found 3 results
2022/09/02 14:24:20 [keyring] Querying keychain for service="aws-vault", keychain="aws-vault.keychain"
2022/09/02 14:24:20 [keyring] Found 3 results
2022/09/02 14:24:20 profile parent: using stored credentials
2022/09/02 14:24:20 profile parent: skipping GetSessionToken because disabled
2022/09/02 14:24:20 profile child: using AssumeRole (with MFA)
2022/09/02 14:24:20 [keyring] Querying keychain for service="aws-vault", keychain="aws-vault.keychain"
2022/09/02 14:24:20 [keyring] Found 3 results
2022/09/02 14:24:20 [keyring] Querying keychain for service="aws-vault", keychain="aws-vault.keychain"
2022/09/02 14:24:20 [keyring] Found 3 results
2022/09/02 14:24:20 [keyring] Querying keychain for service="aws-vault", account="sts.AssumeRole,eW9jby1hd3MtZGV2,YXJuOmF3czppYW06OjA1MjI5NDI1NTQwMjptZmEvYnJ5YW4uaA,-62135596800", keychain="aws-vault.keychain"
2022/09/02 14:24:20 [keyring] No results found
Enter MFA code for arn:aws:iam::111111111111:mfa/bryan.h: 538170
2022/09/02 14:24:23 Looking up keyring for 'parent'
2022/09/02 14:24:23 [keyring] Querying keychain for service="aws-vault", account="parent", keychain="aws-vault.keychain"
2022/09/02 14:24:23 [keyring] Found item "aws-vault (parent)"
aws-vault: error: exec: Failed to get credentials for child: operation error STS: AssumeRole, https response error StatusCode: 403, RequestID: 6b35cb7e-aad7-4e90-aa1f-d97944fc676a, api error AccessDenied: User: arn:aws:iam::111111111111:user/bryan.h is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::222222222222:role/OrganizationAccountAccessRole-Admin

# bryanhorstmann at THVQ46M6RN in ~ [14:24:24]
→

Michael Tibben · Answer 5 · Sat Sep 03 2022 07:10:55 GMT+0800 (China Standard Time)

can you provide the debug output for the successful aws-vault6.3.1 exec

Bryan Horstmann · Answer 6 · Tue Sep 06 2022 23:53:27 GMT+0800 (China Standard Time)

Apologies for the delayed response, this was one of those "saw notification on phone and read it then never actioned" events.

→ aws-vault6.3.1 exec --debug child --no-session -- aws s3 ls
2022/09/06 17:50:58 aws-vault v6.3.1
2022/09/06 17:50:58 Loading config file /Users/bryanhorstmann/.aws/config
2022/09/06 17:50:58 Parsing config file /Users/bryanhorstmann/.aws/config
2022/09/06 17:50:58 [keyring] Considering backends: [keychain]
2022/09/06 17:50:58 [keyring] Querying keychain for service="aws-vault", keychain="aws-vault.keychain"
2022/09/06 17:50:58 [keyring] Found 3 results
2022/09/06 17:50:58 [keyring] Querying keychain for service="aws-vault", keychain="aws-vault.keychain"
2022/09/06 17:50:58 [keyring] Found 3 results
2022/09/06 17:50:58 profile parent: using stored credentials
2022/09/06 17:50:58 profile parent: skipping GetSessionToken because disabled
2022/09/06 17:50:58 profile child: using AssumeRole (with MFA)
2022/09/06 17:50:58 [keyring] Querying keychain for service="aws-vault", keychain="aws-vault.keychain"
2022/09/06 17:50:58 [keyring] Found 3 results
2022/09/06 17:50:58 [keyring] Querying keychain for service="aws-vault", keychain="aws-vault.keychain"
2022/09/06 17:50:58 [keyring] Found 3 results
2022/09/06 17:50:58 [keyring] Querying keychain for service="aws-vault", account="sts.AssumeRole,eW9jby1hd3MtZGV2,YXJuOmF3czppYW06OjA1MjI5NDI1NTQwMjptZmEvYnJ5YW4uaA,-62135596800", keychain="aws-vault.keychain"
2022/09/06 17:50:58 [keyring] No results found
Enter token for arn:aws:iam::111111111111:mfa/bryan.h: 564244
2022/09/06 17:51:05 Using STS endpoint https://sts.amazonaws.com
2022/09/06 17:51:05 Looking up keyring for 'parent'
2022/09/06 17:51:05 [keyring] Querying keychain for service="aws-vault", account="parent", keychain="aws-vault.keychain"
2022/09/06 17:51:05 [keyring] Found item "aws-vault (parent)"
2022/09/06 17:51:06 Generated credentials ****************XXXX using AssumeRole, expires in 1h0m0.274491s
2022/09/06 17:51:06 [keyring] Querying keychain for service="aws-vault", keychain="aws-vault.keychain"
2022/09/06 17:51:06 [keyring] Found 3 results
2022/09/06 17:51:06 [keyring] Querying keychain for service="aws-vault", keychain="aws-vault.keychain"
2022/09/06 17:51:06 [keyring] Found 3 results
2022/09/06 17:51:06 [keyring] Checking keychain status
2022/09/06 17:51:06 [keyring] Keychain status returned nil, keychain exists
2022/09/06 17:51:06 [keyring] Keychain item trusts keyring
2022/09/06 17:51:06 [keyring] Adding service="aws-vault", label="aws-vault session for child (expires 2022-09-06T16:51:07Z)", account="sts.AssumeRole,eW9jby1hd3MtZGV2,YXJuOmF3czppYW06OjA1MjI5NDI1NTQwMjptZmEvYnJ5YW4uaA,1662483067", trusted=true to osx keychain "aws-vault.keychain"
2022/09/06 17:51:06 Setting subprocess env: AWS_DEFAULT_REGION=eu-west-1, AWS_REGION=eu-west-1
2022/09/06 17:51:06 Setting subprocess env: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
2022/09/06 17:51:06 Setting subprocess env: AWS_SESSION_TOKEN, AWS_SECURITY_TOKEN
2022/09/06 17:51:06 Setting subprocess env: AWS_SESSION_EXPIRATION
2022/09/06 17:51:06 Exec command aws s3 ls
2022/09/06 17:51:06 Found executable /usr/local/bin/aws
2021-10-29 09:17:23 bucket1
2020-06-30 10:59:22 bucket2
2020-05-07 10:25:33 bucket3
2021-04-15 11:20:03 bucket4

Bryan Horstmann · Answer 7 · Mon Nov 28 2022 15:49:34 GMT+0800 (China Standard Time)

Hi @mtibben, just following up on this. Any ideas or additional troubleshooting I can try?

Bryan Horstmann · Answer 8 · Wed Feb 01 2023 21:27:14 GMT+0800 (China Standard Time)

Hi @mtibben, just following up on this for guidance

Michael Tibben · Answer 9 · Sun Feb 19 2023 17:15:55 GMT+0800 (China Standard Time)

@bryanhorstmann I note you are using 2 different regions between the parent and child profiles. Perhaps try using the same region?

Can you also trying using the latest v7

Bryan Horstmann · Answer 10 · Tue Apr 04 2023 20:38:16 GMT+0800 (China Standard Time)

Hi @mtibben ,

Apologies for the delayed response. We figured this out by complete accident. As a team, we realised that it was only members of our Administrator group that could not assume roles. This group had no explicit assume role policy assigned, but was relying on blanket wildcard permissions to work. Once we assigned relevant assume role policies to the Administrator group it worked. This includes the latest v7.

Thank you for your assistance.