NoCredentialProviders: no valid providers in chain when adding a certain aws profile

Question

NoCredentialProviders: no valid providers in chain when adding a certain aws profile

HenryXie1 opened this issue a year ago · comments

Describe the bug

We have blow code to describe eks cluster

sess, err := session.NewSessionWithOptions(session.Options{
		Config:            aws.Config{Region: aws.String("ap-southeast-2"), CredentialsChainVerboseErrors: aws.Bool(true)},
		SharedConfigFiles: []string{awsConfigPath},
		SharedConfigState: session.SharedConfigEnable,
		Profile:           clusterName,
	})
	if err != nil {
		fmt.Println("clustername: " + clusterName)
		return err
	}

	fmt.Println("new cluster session good: " + clusterName)
	eksClient := eks.New(sess)
	cluster, err := eksClient.DescribeCluster(&eks.DescribeClusterInput{Name: &clusterName})

	if err != nil {
		return err
	}

it works well before.
after we add below line in the ./aws/config , it starts error out

[profile e02-admin]
sso_start_url = ******
role_arn = arn:aws:iam::******:role/AdministratorRole
source_profile = e02
sso_role_name = PlatformAdmin
region = ap-southeast-2

The error is

new cluster session good: e02
 NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, e02.
EC2RoleRequestError: no EC2 instance role found
caused by: RequestError: send request failed
caused by: Get "[http://169.254.169.254/latest/meta-data/iam/security-credentials/ ](http://169.254.169.254/latest/meta-data/iam/security-credentials/)": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Expected Behavior

it was working well before we add e02-admin profile.

Current Behavior

it error out after we add this profile in the .aws/config

[profile e02-admin]
sso_start_url = ******
role_arn = arn:aws:iam::******:role/AdministratorRole
source_profile = e02
sso_role_name = PlatformAdmin
region = ap-southeast-2

profile e02 exists before this issue happens

[profile e02]
sso_start_url = ***
sso_region = ap-southeast-2
sso_account_id = ***
sso_role_name = PlatformAdmin
region = ap-southeast-2

Reproduction Steps

Add e02-admin profile in .aws/config
then the issue is reproduced

Possible Solution

change source_profile in e02-admin from e02 to others (ie d02)
or
remove e02-admin from .aws/config
the issue is gone

Additional Information/Context

Go verison

go 1.19

require (
	github.com/aws/aws-sdk-go v1.45.9
	github.com/awslabs/goformation/v4 v4.19.5
	github.com/buger/jsonparser v1.1.1
	github.com/urfave/cli v1.22.5
	gopkg.in/yaml.v2 v2.4.0
	k8s.io/api v0.25.3
	k8s.io/apimachinery v0.25.3
	k8s.io/client-go v0.25.3
)

SDK version used

1.45.9

Environment details (Version of Go (`go version`)? OS name and version, etc.)

go version go1.19.1 darwin/amd64

Henry Xie · Answer 1 · Fri Sep 15 2023 05:54:42 GMT+0800 (China Standard Time)

the aws cli is working well with the e02-admin and e02 profiles

❯ aws eks update-kubeconfig --profile e02-admin --name e02
Updated context arn:aws:eks:ap-southeast-2:***:cluster/e02 in /Users/xieho/.kube/config

~/Documents
❯ aws s3 ls --profile e02-admin
2023-07-06 08:46:41 test
2023-08-23 14:45:16 test1
2022-08-26 15:00:20 test2

aws eks describe-cluster --name e02  --profile e02-admin
...
aws eks describe-cluster --name e02  --profile e02
...

Henry Xie · Answer 2 · Mon Sep 18 2023 05:17:51 GMT+0800 (China Standard Time)

I enabled LogLevel: aws.LogLevel(aws.LogDebugWithHTTPBody) in the code, find below
trace

2023/09/18 07:02:01 DEBUG: Request ec2metadata/GetToken Details:
---[ REQUEST POST-SIGN ]-----------------------------
PUT /latest/api/token HTTP/1.1
Host: 169.254.169.254
User-Agent: aws-sdk-go/1.45.9 (go1.19.1; darwin; amd64)
Content-Length: 0
X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21600
Accept-Encoding: gzip
....

After I implemented workaround, change source_profile from e02 to d02
the issue is gone. The trace is like

2023/09/18 07:03:02 DEBUG: Request SSO/GetRoleCredentials Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /federation/credentials?account_id=1234***&role_name=PlatformAdmin HTTP/1.1
Host: portal.sso.ap-southeast-2.amazonaws.com
User-Agent: aws-sdk-go/1.45.9 (go1.19.1; darwin; amd64)
X-Amz-Sso_bearer_token: *****
Accept-Encoding: gzip

Because I use sso federation in aws profile, I wonder why sdk is looking for ec2metadata/GetToken in this situation.

Henry Xie · Answer 3 · Mon Sep 18 2023 05:21:13 GMT+0800 (China Standard Time)

We tested on d02 profile which has same content as e02, it is working well.

In this case, it seems that the SDK is not correctly recognizing the SSO federation credentials only from e02 profile, causing it to fall back to the EC2 metadata provider.

but AWS CLI is working fine with the same profiles, this might be a bug in the SDK
Wonder anything special with the key word e02 in aws sdk go?

Ran Vaknin · Answer 4 · Thu Sep 21 2023 02:38:37 GMT+0800 (China Standard Time)

Hi @HenryXie1 ,

Thanks for all the details, this is indeed an interesting behavior.

If IRSA credentials fail, the logs would typically show a failed request to the OIDC identity provider and then to the STS AssumeRoleWithWebIdentity operation. Errors could include token validation issues, service errors, etc. Are you able to see any logs of errors with that credentials provider?

Another thought I have was for you to set the desired profile explicitly in the SDK and see if that helps with resolving the issue, or at least would surface some more helpful errors. You can check out this blogpost .

Let us know what you find.
Thanks,
Ran~

Henry Xie · Answer 5 · Thu Sep 21 2023 14:12:11 GMT+0800 (China Standard Time)

Hey @RanVaknin
Thanks for your reply
Here are some input I have

Are you able to see any logs of errors with that credentials provider?

no, do you know how to enable some debug trace for it?

Another thought I have was for you to set the desired profile explicitly in the SDK and see if that helps with resolving the issue, or at least would surface some more helpful errors. You can check out this blogpost .

I add below code to debug, but i don't see any output from sts client and error is also nil

client := sts.New(sess)
	fmt.Println("sts client info2:")
	fmt.Println("sts client:", *client)
	identity, err := client.GetCallerIdentity(&sts.GetCallerIdentityInput{})
	fmt.Println("sts client info3:")
	fmt.Println("sts identity:", *identity)

    if err != nil {
        fmt.Println("error:", err)
        os.Exit(1)
    }

    fmt.Printf(
        "Account: %s\nUserID: %s\nARN: %s\n",
        aws.StringValue(identity.Account),
        aws.StringValue(identity.UserId),
        aws.StringValue(identity.Arn),
    )

sts client info1:
sts client info2:
sts client: {0xc000423c00}
2023/09/21 16:03:10 DEBUG: Request ec2metadata/GetToken Details:
---[ REQUEST POST-SIGN ]-----------------------------
PUT /latest/api/token HTTP/1.1
Host: 169.254.169.254
User-Agent: aws-sdk-go/1.45.9 (go1.19.1; darwin; amd64)
Content-Length: 0
X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21600
Accept-Encoding: gzip

After I change source_profile in e02-admin from e02 to d02(workaround)
The output is

❯ ~/github-cba/pksctl/bin/pksctl generate kube-config
sts client info1:
sts client info2:
sts client: {0xc000411c00}
2023/09/21 16:05:49 DEBUG: Request SSO/GetRoleCredentials Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /federation/credentials?account_id=****&role_name=PlatformAdmin HTTP/1.1
Host: portal.sso.ap-southeast-2.amazonaws.com
User-Agent: aws-sdk-go/1.45.9 (go1.19.1; darwin; amd64)
X-Amz-Sso_bearer_token: a****
Accept-Encoding: gzip

It is weird that i don't see println for sts client info3

Ran Vaknin · Answer 6 · Fri Sep 22 2023 04:45:09 GMT+0800 (China Standard Time)

Hi @HenryXie1,

We think this is an issue with how the SDK's INI loader parses numbers. @lucix-aws has raised a PR regarding a similar issue we saw on v2, and tested the ini parser with d02 as an argument without an issue, and then e02 and we saw an issue. It might be related.
Can you please wait for tomorrow's release, pull the latest version of the SDK, and see if this solves your issue?

Thanks,
Ran~

Luc Talatinian · Answer 7 · Sat Sep 23 2023 03:48:00 GMT+0800 (China Standard Time)

@HenryXie1 --

Pursuant to @RanVaknin's comment above, release 1.45.15 has just been tagged, which should solve your issue.

Henry Xie · Answer 8 · Sat Sep 23 2023 04:36:11 GMT+0800 (China Standard Time)

Great, thanks @lucix-aws @RanVaknin

Luc Talatinian · Answer 9 · Sun Sep 24 2023 04:42:29 GMT+0800 (China Standard Time)

Please let us know if that unblocks you so we can close this out or investigate further.

Henry Xie · Answer 10 · Mon Sep 25 2023 03:23:40 GMT+0800 (China Standard Time)

@lucix-aws I confirm the issue is fixed by upgrading aws go sdk to 1.45.15

github-actions · Answer 11 · Mon Sep 25 2023 07:39:51 GMT+0800 (China Standard Time)

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

NoCredentialProviders: no valid providers in chain when adding a certain aws profile

Describe the bug

Expected Behavior

Current Behavior

Reproduction Steps

Possible Solution

Additional Information/Context

SDK version used

Environment details (Version of Go (go version)? OS name and version, etc.)

⚠️COMMENT VISIBILITY WARNING⚠️

Environment details (Version of Go (`go version`)? OS name and version, etc.)