SSO broken in aws-vault 7.1.0

Question

SSO broken in aws-vault 7.1.0

brontolinux opened this issue a year ago · comments

I am using the latest release of AWS Vault
I have provided my .aws/config (redacted if necessary)
I have provided the debug output using aws-vault --debug (redacted if necessary)

I have a set-up of aws-vault with AWS SSO (IAM Identity Center). Things worked perfectly with v7.0.x and 7.0.1 in particular.

This morning I upgraded to 7.1.0. As soon as I tried to either access the console or a shell, I got a failure:

$ aws-vault login myprofile 
aws-vault: error: login: Failed to get credentials: operation error STS: GetSessionToken, https response error StatusCode: 403, RequestID: 06d17f46-6d61-4801-a0bf-9015c3ee5bf9, api error InvalidClientTokenId: The security token included in the request is invalid.

Same thing with exec:

$ aws-vault exec myprofile 
aws-vault: error: exec: Failed to get credentials for rikstvtest: operation error STS: GetSessionToken, https response error StatusCode: 403, RequestID: 7d4077d7-3f3e-4399-b6c1-cc07aa723478, api error InvalidClientTokenId: The security token included in the request is invalid.

The relevant part of config is (redacted):

[profile myprofile]
sso_start_url=https://mycompany.awsapps.com/start
sso_region=eu-west-1
sso_account_id=1234567890
sso_role_name=SRE-Admin

One thing that may be relevant is that, before using AWS SSO, the profile myprofile had "static" credentials, that we rotated every 24 hours and I added to my laptop's keyring with aws-vault add. It could be that aws-vault 7.1.0 is pointing to those expired credentials instead of running the SSO authentication, but I don't know if it's the case.

I am using Debian Linux 11 with the latest security updates. My desktop environment is GNOME.

HTH

Ciao
-- bronto

Michael Tibben · Answer 1 · Tue Mar 14 2023 18:47:06 GMT+0800 (China Standard Time)

Please provide debug output

Marco Marongiu · Answer 2 · Wed Mar 15 2023 23:10:40 GMT+0800 (China Standard Time)

With 7.1.0 (doesn't work):

$ aws-vault-7.1.0 --debug login myprofile
2023/03/15 16:01:36 aws-vault v7.1.0
2023/03/15 16:01:36 Using prompt driver: terminal
2023/03/15 16:01:36 [keyring] Considering backends: [secret-service]
2023/03/15 16:01:36 Loading config file /home/bronto/.aws/config
2023/03/15 16:01:36 Parsing config file /home/bronto/.aws/config
2023/03/15 16:01:36 Using region "eu-west-1" from AWS_DEFAULT_REGION
2023/03/15 16:01:36 Profile 'default' missing in config file
2023/03/15 16:01:36 profile myprofile: using stored credentials
2023/03/15 16:01:36 profile myprofile: using GetSessionToken 
2023/03/15 16:01:36 Looking up keyring for 'myprofile'
aws-vault: error: login: Failed to get credentials: operation error STS: GetSessionToken, https response error StatusCode: 403, RequestID: c5e349d3-8075-4eaa-941d-d44c6ff005dd, api error InvalidClientTokenId: The security token included in the request is invalid.

$  /var/cfengine/cache/bin/aws-vault-7.1.0 --debug exec myprofile
2023/03/15 16:02:43 aws-vault v7.1.0
2023/03/15 16:02:43 Using prompt driver: terminal
2023/03/15 16:02:43 Loading config file /home/bronto/.aws/config
2023/03/15 16:02:43 Parsing config file /home/bronto/.aws/config
2023/03/15 16:02:43 [keyring] Considering backends: [secret-service]
2023/03/15 16:02:43 Using region "eu-west-1" from AWS_DEFAULT_REGION
2023/03/15 16:02:43 Profile 'default' missing in config file
2023/03/15 16:02:43 profile myprofile: using stored credentials
2023/03/15 16:02:43 profile myprofile: using GetSessionToken 
2023/03/15 16:02:43 Setting subprocess env: AWS_REGION=eu-west-1, AWS_DEFAULT_REGION=eu-west-1
2023/03/15 16:02:43 Looking up keyring for 'myprofile'
aws-vault: error: exec: Failed to get credentials for myprofile: operation error STS: GetSessionToken, https response error StatusCode: 403, RequestID: 278ed1fb-b1ef-41a5-8cc6-329fa111b540, api error InvalidClientTokenId: The security token included in the request is invalid.

With 7.0.1 (works):

$ aws-vault --debug exec myprofile
2023/03/15 16:04:26 aws-vault v7.0.1
2023/03/15 16:04:26 Using prompt driver: terminal
2023/03/15 16:04:26 Loading config file /home/bronto/.aws/config
2023/03/15 16:04:26 Parsing config file /home/bronto/.aws/config
2023/03/15 16:04:26 [keyring] Considering backends: [secret-service]
2023/03/15 16:04:26 Using region "eu-west-1" from AWS_DEFAULT_REGION
2023/03/15 16:04:26 Profile 'default' missing in config file
2023/03/15 16:04:26 profile myprofile: using SSO role credentials
2023/03/15 16:04:26 Setting subprocess env: AWS_REGION=eu-west-1, AWS_DEFAULT_REGION=eu-west-1
2023/03/15 16:04:26 Created new OIDC client (expires at: 2023-06-13 17:04:26 +0200 CEST)
2023/03/15 16:04:26 Created OIDC device code for https://mycompany.awsapps.com/start (expires in: 600s)
2023/03/15 16:04:26 Opening SSO authorization page in browser
Opening the SSO authorization page in your default browser (use Ctrl-C to abort)
https://device.sso.eu-west-1.amazonaws.com/?user_code=XXXX-XXXX
2023/03/15 16:04:33 Created new OIDC access token for https://mycompany.awsapps.com/start (expires in: 11428s)
2023/03/15 16:04:33 Got credentials ****************6QKT for SSO role SRE-Admin (account: 1234567890), expires in 7h59m58.628982653s
2023/03/15 16:04:33 Setting subprocess env: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
2023/03/15 16:04:33 Setting subprocess env: AWS_SESSION_TOKEN
2023/03/15 16:04:33 Setting subprocess env: AWS_CREDENTIAL_EXPIRATION
2023/03/15 16:04:33 Starting subshell /bin/bash, use `exit` to exit the subshell
2023/03/15 16:04:33 Exec command /bin/bash 
2023/03/15 16:04:33 Found executable /bin/bash

So it definitely looks like a regression to me. 7.1.0 gets to the old, expired credentials from the secret-service, tries to use those, and fails. 7.0.1 considers secret-service, too, but then finds the SSO configuration in myprofile and uses that one instead.

Fernando Miguel · Answer 3 · Thu Mar 16 2023 00:11:48 GMT+0800 (China Standard Time)

as a data point, i'm not seeing this issue on v7.1.0 and using SSO to tens of accounts

Marco Marongiu · Answer 4 · Thu Mar 16 2023 06:15:24 GMT+0800 (China Standard Time)

as a data point, i'm not seeing this issue on v7.1.0 and using SSO to tens of accounts

Hi @FernandoMiguel

What makes a difference in my case is that, for that profile, I have some old static credentials in the keyring that were in use before I switched that profile to SSO. If I didn't have those, 7.1.0 would work just fine: it would not find credentials in secret-service that match the requested profile and would use config instead, so SSO. But 7.1.0 finds the old static credentials and tries to use them instead of relying on the authoritative information in config, that indicates SSO as the source of credentials.

7.0.1 does the right thing there, in that it finds credentials through both secret-service and config, and uses the latter. That's why I think this is a regression.

Michael Tibben · Answer 5 · Thu Mar 16 2023 07:05:56 GMT+0800 (China Standard Time)

Yes it looks like your myprofile is using stored credentials instead of the sso config. You can aws-vault rm myprofile to remove the old credentials. v7.0.0 and v7.0.1 preferenced the SSO config incorrectly, fixed in v7.0.2.

Marco Marongiu · Answer 6 · Thu Mar 16 2023 17:02:01 GMT+0800 (China Standard Time)

Yes it looks like your myprofile is using stored credentials instead of the sso config. You can aws-vault rm myprofile to remove the old credentials. v7.0.0 and v7.0.1 preferenced the SSO config incorrectly, fixed in v7.0.2.

Hi @mtibben
I can confirm that everything worked in 7.1.0 after removing the old static credentials.
I feel that the "wrong" behaviour in v7.0.1 was more consistent in that it considered config to be authoritative over the rest, but... aws-vault is yours, and so the decision 😉
Ciao
-- bronto

Michael Tibben · Answer 7 · Thu Mar 16 2023 17:35:11 GMT+0800 (China Standard Time)

I don't disagree @brontolinux.... I also thought this behaviour would be more correct but ended up with many side-effects.... see #1183