Doesn't recognize AWS_WEB_IDENTITY_TOKEN_FILE, botocore.exceptions.ProfileNotFound: The config profile (default) could not be found

Question

Doesn't recognize AWS_WEB_IDENTITY_TOKEN_FILE, botocore.exceptions.ProfileNotFound: The config profile (default) could not be found

rfvermut opened this issue 3 years ago · comments

Pavels Veretennikovs commented 3 years ago

In AWS EKS environment with IRSA. In version 0.22.

AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token

awscurl -X PUT https://example.com
Traceback (most recent call last):
  File "/usr/local/bin/awscurl", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/awscurl/awscurl.py", line 504, in main
    inner_main(sys.argv[1:])
  File "/usr/local/lib/python3.9/site-packages/awscurl/awscurl.py", line 469, in inner_main
    args.access_key, args.secret_key, args.session_token = load_aws_config(args.access_key,
  File "/usr/local/lib/python3.9/site-packages/awscurl/awscurl.py", line 393, in load_aws_config
    cred = session.get_credentials()
  File "/usr/local/lib/python3.9/site-packages/botocore/session.py", line 441, in get_credentials
    self._credentials = self._components.get_component(
  File "/usr/local/lib/python3.9/site-packages/botocore/session.py", line 937, in get_component
    self._components[name] = factory()
  File "/usr/local/lib/python3.9/site-packages/botocore/session.py", line 151, in _create_credential_resolver
    return botocore.credentials.create_credential_resolver(
  File "/usr/local/lib/python3.9/site-packages/botocore/credentials.py", line 64, in create_credential_resolver
    metadata_timeout = session.get_config_variable('metadata_service_timeout')
  File "/usr/local/lib/python3.9/site-packages/botocore/session.py", line 251, in get_config_variable
    return self.get_component('config_store').get_config_variable(
  File "/usr/local/lib/python3.9/site-packages/botocore/configprovider.py", line 313, in get_config_variable
    return provider.provide()
  File "/usr/local/lib/python3.9/site-packages/botocore/configprovider.py", line 410, in provide
    value = provider.provide()
  File "/usr/local/lib/python3.9/site-packages/botocore/configprovider.py", line 471, in provide
    scoped_config = self._session.get_scoped_config()
  File "/usr/local/lib/python3.9/site-packages/botocore/session.py", line 351, in get_scoped_config
    raise ProfileNotFound(profile=profile_name)
botocore.exceptions.ProfileNotFound: The config profile (default) could not be found

Works fine in 0.21

Igor Okulist · Answer 1 · Sat Jul 03 2021 14:33:43 GMT+0800 (China Standard Time)

Thanks for reporting - I’ll have a look

Iain Samuel McLean Elder · Answer 2 · Sat Jul 03 2021 19:55:00 GMT+0800 (China Standard Time)

@rfvermut How are your other CLI environment variables set?

https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html

What sections do you have in your CLI config file and credentials file?

https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html

Pavels Veretennikovs · Answer 3 · Sat Jul 03 2021 22:37:35 GMT+0800 (China Standard Time)

Everything is completely empty, no configuration. Docker image python:3-buster, pip3 install awscli awscurl

Iain Samuel McLean Elder · Answer 4 · Sun Jul 04 2021 00:11:03 GMT+0800 (China Standard Time)

I'm unfamiliar with AWS EKS. Does IRSA refer to the IAM roles for service accounts feature?

I don't think the behavior is affecterd by AWS_WEB_IDENTITY_TOKEN_FILE so I'll ignore it for now.

The server at example.com doesn't appear to respond to PUT requests so lets change that to a GET request.

awscurl -X GET https://example.com

So here's what I think is happening.

The behavior occurs only when botocore is available to awscurl.

Without knowing more about your environment, I will be using pipx to install awscurl in virtualenv and use the awslibs optional dependency to ensure that botocore is available.

pipx install awscurl[awslibs]==0.22.0

With the example command, awscurl via botocore loads the "default" profile (because that's the default value set by the CLI parser The default value is not None as I had previously thought!).

Since you have no configuration, botocore fails to find that profile and so fails.

But what happens in the previous version?

pipx uninstall awscurl
pipx install awscurl[awslibs]==0.21.0

With the example command, on my local machine, I get a different error, but it's for the same reason; there are no credentials to be found.

$ awscurl -X PUT https://example.com
Traceback (most recent call last):
  File "/home/norm/.local/bin/awscurl", line 8, in <module>
    sys.exit(main())
  File "/home/norm/.local/pipx/venvs/awscurl/lib/python3.8/site-packages/awscurl/awscurl.py", line 500, in main
    inner_main(sys.argv[1:])
  File "/home/norm/.local/pipx/venvs/awscurl/lib/python3.8/site-packages/awscurl/awscurl.py", line 466, in inner_main
    args.access_key, args.secret_key, args.session_token = load_aws_config(args.access_key,
  File "/home/norm/.local/pipx/venvs/awscurl/lib/python3.8/site-packages/awscurl/awscurl.py", line 392, in load_aws_config
    access_key, secret_key, security_token = cred.access_key, cred.secret_key, cred.token
AttributeError: 'NoneType' object has no attribute 'access_key'

If I execute it on an EC2 instance with an attached IAM role, then awscurl via botocore loads the instance credentials from the instance metadata service.

@okigan I looks like this behavior was broken by my PR #116 for AWS SSO profiles.

I had believed that the default value for profile was None, but actually it's hard-coded to "default". That changes how botocore's default credential resolution works. It stops it falling back to the instance metadata servce.

I think there is a way to support all the desired credentials sources, but it will take time to find a correct solution.

For now I think the right thing to do is to undo my changes and not include them until we find a way to test all the use cases.

Iain Samuel McLean Elder · Answer 5 · Sun Jul 04 2021 00:12:33 GMT+0800 (China Standard Time)

@rfvermut I'm not aware of a way to workaround this with version 0.22.0. For now I suggest you specify 0.21.0 when you install the command.

Igor Okulist · Answer 6 · Sun Jul 04 2021 00:35:42 GMT+0800 (China Standard Time)

I'm unfamiliar with AWS EKS. Does IRSA refer to the IAM roles for service accounts feature?

I don't think the behavior is affecterd by AWS_WEB_IDENTITY_TOKEN_FILE so I'll ignore it for now.

The server at example.com doesn't appear to respond to PUT requests so lets change that to a GET request.
awscurl -X GET https://example.com
So here's what I think is happening.

The behavior occurs only when botocore is available to awscurl.

Without knowing more about your environment, I will be using pipx to install awscurl in virtualenv and use the awslibs optional dependency to ensure that botocore is available.
pipx install awscurl[awslibs]==0.22.0
With the example command, awscurl via botocore loads the "default" profile (because that's the default value set by the CLI parser The default value is not None as I had previously thought!).

Since you have no configuration, botocore fails to find that profile and so fails.

But what happens in the previous version?
pipx uninstall awscurl
pipx install awscurl[awslibs]==0.21.0
With the example command, on my local machine, I get a different error, but it's for the same reason; there are no credentials to be found.
$ awscurl -X PUT https://example.com
Traceback (most recent call last):
  File "/home/norm/.local/bin/awscurl", line 8, in <module>
    sys.exit(main())
  File "/home/norm/.local/pipx/venvs/awscurl/lib/python3.8/site-packages/awscurl/awscurl.py", line 500, in main
    inner_main(sys.argv[1:])
  File "/home/norm/.local/pipx/venvs/awscurl/lib/python3.8/site-packages/awscurl/awscurl.py", line 466, in inner_main
    args.access_key, args.secret_key, args.session_token = load_aws_config(args.access_key,
  File "/home/norm/.local/pipx/venvs/awscurl/lib/python3.8/site-packages/awscurl/awscurl.py", line 392, in load_aws_config
    access_key, secret_key, security_token = cred.access_key, cred.secret_key, cred.token
AttributeError: 'NoneType' object has no attribute 'access_key'
If I execute it on an EC2 instance with an attached IAM role, then awscurl via botocore loads the instance credentials from the instance metadata service.

@okigan I looks like this behavior was broken by my PR #116 for AWS SSO profiles.

I had believed that the default value for profile was None, but actually it's hard-coded to "default". That changes how botocore's default credential resolution works. It stops it falling back to the instance metadata servce.

I think there is a way to support all the desired credentials sources, but it will take time to find a correct solution.

For now I think the right thing to do is to undo my changes and not include them until we find a way to test all the use cases.

@iainelder i will be backing out the SSO feature/commit and releasing bug fix build - that should buy us time to figure out what’s going on here.

Igor Okulist · Answer 7 · Sun Jul 04 2021 05:45:24 GMT+0800 (China Standard Time)

Change reverted: #124, new release will follow shortly

Igor Okulist · Answer 8 · Sun Jul 04 2021 07:15:30 GMT+0800 (China Standard Time)

@rfvermut I am looking for a way to setup a test environment for this, is the following script sufficient to repo? and in which environment does that need to run?

docker run  -it python:3-buster /bin/bash -c "pip3 install awscli awscurl && awscurl -X GET https://example.com"

Pavels Veretennikovs · Answer 9 · Sun Jul 04 2021 15:30:00 GMT+0800 (China Standard Time)

First, let me thank you reverting that change so fast. I have a gazillion non-centralized Jenkins scripts that refer to non-pinned install of awscurl and it affected half of our organization. My mistake, always pin your versions.

Second, the "just enough" setup for tests would look like this:

Get inside a container

docker run -ti --rm python:3-buster /bin/bash

Prepare environment

export AWS_ROLE_ARN="arn:aws:iam::123456789012:role/marketingadminrole"
export AWS_WEB_IDENTITY_TOKEN_FILE="/tmp/test"
echo "garbage" > $AWS_WEB_IDENTITY_TOKEN_FILE
pip3 install awscli awscurl==0.23 (or 22)

0.23/0.21 will fail trying to decode garbage while doing AssumeRoleWithWebIdentity which means it understands identity tokens

root@369514c4e8e8:/# awscurl example.com
Traceback (most recent call last):
  File "/usr/local/bin/awscurl", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/awscurl/awscurl.py", line 504, in main
    inner_main(sys.argv[1:])
  File "/usr/local/lib/python3.9/site-packages/awscurl/awscurl.py", line 469, in inner_main
    args.access_key, args.secret_key, args.session_token = load_aws_config(args.access_key,
  File "/usr/local/lib/python3.9/site-packages/awscurl/awscurl.py", line 394, in load_aws_config
    access_key, secret_key, security_token = cred.access_key, cred.secret_key, cred.token
  File "/usr/local/lib/python3.9/site-packages/botocore/credentials.py", line 421, in access_key
    self._refresh()
  File "/usr/local/lib/python3.9/site-packages/botocore/credentials.py", line 513, in _refresh
    self._protected_refresh(is_mandatory=is_mandatory_refresh)
  File "/usr/local/lib/python3.9/site-packages/botocore/credentials.py", line 529, in _protected_refresh
    metadata = self._refresh_using()
  File "/usr/local/lib/python3.9/site-packages/botocore/credentials.py", line 670, in fetch_credentials
    return self._get_cached_credentials()
  File "/usr/local/lib/python3.9/site-packages/botocore/credentials.py", line 680, in _get_cached_credentials
    response = self._get_credentials()
  File "/usr/local/lib/python3.9/site-packages/botocore/credentials.py", line 890, in _get_credentials
    return client.assume_role_with_web_identity(**kwargs)
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 386, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 705, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.InvalidIdentityTokenException: An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation: The ID Token provided is not a valid JWT. (You may see this error if you sent an Access Token)

0.22 will die as usual.

root@369514c4e8e8:/# awscurl example.com
Traceback (most recent call last):
  File "/usr/local/bin/awscurl", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/awscurl/awscurl.py", line 504, in main
    inner_main(sys.argv[1:])
  File "/usr/local/lib/python3.9/site-packages/awscurl/awscurl.py", line 469, in inner_main
    args.access_key, args.secret_key, args.session_token = load_aws_config(args.access_key,
  File "/usr/local/lib/python3.9/site-packages/awscurl/awscurl.py", line 393, in load_aws_config
    cred = session.get_credentials()
  File "/usr/local/lib/python3.9/site-packages/botocore/session.py", line 441, in get_credentials
    self._credentials = self._components.get_component(
  File "/usr/local/lib/python3.9/site-packages/botocore/session.py", line 937, in get_component
    self._components[name] = factory()
  File "/usr/local/lib/python3.9/site-packages/botocore/session.py", line 151, in _create_credential_resolver
    return botocore.credentials.create_credential_resolver(
  File "/usr/local/lib/python3.9/site-packages/botocore/credentials.py", line 64, in create_credential_resolver
    metadata_timeout = session.get_config_variable('metadata_service_timeout')
  File "/usr/local/lib/python3.9/site-packages/botocore/session.py", line 251, in get_config_variable
    return self.get_component('config_store').get_config_variable(
  File "/usr/local/lib/python3.9/site-packages/botocore/configprovider.py", line 313, in get_config_variable
    return provider.provide()
  File "/usr/local/lib/python3.9/site-packages/botocore/configprovider.py", line 410, in provide
    value = provider.provide()
  File "/usr/local/lib/python3.9/site-packages/botocore/configprovider.py", line 471, in provide
    scoped_config = self._session.get_scoped_config()
  File "/usr/local/lib/python3.9/site-packages/botocore/session.py", line 351, in get_scoped_config
    raise ProfileNotFound(profile=profile_name)
botocore.exceptions.ProfileNotFound: The config profile (default) could not be found

Igor Okulist · Answer 10 · Tue Jul 06 2021 03:07:51 GMT+0800 (China Standard Time)

@rfvermut thanks for repro script!
@iainelder I have an idea, why profile is not working: in the command line parser it is the only one (among AWS_* related variables) that has a default value (of 'default'), so I think removing that and adding (back) the Session(profile=profile) could make it work. Here is related change: a501a43

Iain Samuel McLean Elder · Answer 11 · Wed Jul 07 2021 01:52:30 GMT+0800 (China Standard Time)

Commit a501a43 looks to me like it could work.

(It's what I was alluding to in my ending comments in #123 (comment))

How would you test it?

Igor Okulist · Answer 12 · Wed Jul 07 2021 02:48:07 GMT+0800 (China Standard Time)

@iainelder i've tested it by checking out and running a501a43.
Getting the InvalidIdentityTokenException error as shown in #122 (comment) using following script:

export AWS_ROLE_ARN="arn:aws:iam::123456789012:role/marketingadminrole"
export AWS_WEB_IDENTITY_TOKEN_FILE="/tmp/test"
echo "garbage" > $AWS_WEB_IDENTITY_TOKEN_FILE
python -m awscurl

i am still looking for a way to test load_aws_config, but do not see a way to mock the relevant component.

Iain Samuel McLean Elder · Answer 13 · Fri Jul 09 2021 18:54:08 GMT+0800 (China Standard Time)

@okigan have you looked at moto? it has mocks for many AWS services including STS.

We might be able to use it to mock responses from the STS service that we are authenticating against.

https://github.com/spulec/moto

I have some ideas here, but no time to implement them for a while yet :-)

Adam Stracener · Answer 14 · Wed Mar 13 2024 02:43:59 GMT+0800 (China Standard Time)

This is still an issue.

Igor Okulist · Answer 15 · Thu Mar 14 2024 22:07:40 GMT+0800 (China Standard Time)

Hi y'all, I don't have enough context on this issue which makes it tricky for me to troubleshoot this at this time. I think having an endpoint and a working example (with other tools) could help. Thanks! Igor

…

On Tue, Mar 12, 2024 at 1:44 PM Adam Stracener ***@***.***> wrote: This is still an issue. — Reply to this email directly, view it on GitHub <#122 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADUYXV2OJHEFCETDMW4HV3YX5EIJAVCNFSM47WN54R2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOJZGIZTEMBTGEYA> . You are receiving this because you were mentioned.Message ID: ***@***.***>