Credentials issue introduced in 0.27.0 when running on EC2 instance.

Question

Credentials issue introduced in 0.27.0 when running on EC2 instance.

mjharvey opened this issue a year ago · comments

Hi. I have been using awscurl on an EC2 instance to call an API endpoint that is secured via IAM permissions.
This worked fine under 0.26.0, but with the release of 0.27.0 I am now encountering this error

  File "/usr/local/bin/awscurl", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/awscurl/awscurl.py", line 543, in main
    inner_main(sys.argv[1:])
  File "/usr/local/lib/python3.6/site-packages/awscurl/awscurl.py", line 508, in inner_main
    args.profile)
  File "/usr/local/lib/python3.6/site-packages/awscurl/awscurl.py", line 427, in load_aws_config
    cred = session.get_credentials()
  File "/usr/local/lib/python3.6/site-packages/botocore/session.py", line 449, in get_credentials
    'credential_provider').load_credentials()
  File "/usr/local/lib/python3.6/site-packages/botocore/session.py", line 897, in get_component
    self._components[name] = factory()
  File "/usr/local/lib/python3.6/site-packages/botocore/session.py", line 176, in <lambda>
    lambda:  botocore.credentials.create_credential_resolver(self))
  File "/usr/local/lib/python3.6/site-packages/botocore/credentials.py", line 55, in create_credential_resolver
    metadata_timeout = session.get_config_variable('metadata_service_timeout')
  File "/usr/local/lib/python3.6/site-packages/botocore/session.py", line 265, in get_config_variable
    elif self._found_in_config_file(methods, var_config):
  File "/usr/local/lib/python3.6/site-packages/botocore/session.py", line 286, in _found_in_config_file
    return var_config[0] in self.get_scoped_config()
  File "/usr/local/lib/python3.6/site-packages/botocore/session.py", line 358, in get_scoped_config
    raise ProfileNotFound(profile=profile_name)
botocore.exceptions.ProfileNotFound: The config profile (default) could not be found

The problem looks like it might have been caused by this change: 7b38c7f?diff=split?diff=split?diff=split%3Fdiff%3Dsplit

Is this a known issue, or this there a workaround for the problem I could apply?

Thanks.

Igor Okulist · Answer 1 · Wed May 03 2023 05:01:29 GMT+0800 (China Standard Time)

Let me know how you invoke awscurl, and also how do you specify the credentials (environment variable, command line, etc). And thanks for reporting the issue.

…

On Tue, May 2, 2023 at 3:35 PM Matt Harvey ***@***.***> wrote: Hi. I have been using awscurl on an EC2 instance to call an API endpoint that is secured via IAM permissions. This worked fine under 0.26.0, but with the release of 0.27.0 I am now encountering this error File "/usr/local/bin/awscurl", line 11, in <module> sys.exit(main()) File "/usr/local/lib/python3.6/site-packages/awscurl/awscurl.py", line 543, in main inner_main(sys.argv[1:]) File "/usr/local/lib/python3.6/site-packages/awscurl/awscurl.py", line 508, in inner_main args.profile) File "/usr/local/lib/python3.6/site-packages/awscurl/awscurl.py", line 427, in load_aws_config cred = session.get_credentials() File "/usr/local/lib/python3.6/site-packages/botocore/session.py", line 449, in get_credentials 'credential_provider').load_credentials() File "/usr/local/lib/python3.6/site-packages/botocore/session.py", line 897, in get_component self._components[name] = factory() File "/usr/local/lib/python3.6/site-packages/botocore/session.py", line 176, in <lambda> lambda: botocore.credentials.create_credential_resolver(self)) File "/usr/local/lib/python3.6/site-packages/botocore/credentials.py", line 55, in create_credential_resolver metadata_timeout = session.get_config_variable('metadata_service_timeout') File "/usr/local/lib/python3.6/site-packages/botocore/session.py", line 265, in get_config_variable elif self._found_in_config_file(methods, var_config): File "/usr/local/lib/python3.6/site-packages/botocore/session.py", line 286, in _found_in_config_file return var_config[0] in self.get_scoped_config() File "/usr/local/lib/python3.6/site-packages/botocore/session.py", line 358, in get_scoped_config raise ProfileNotFound(profile=profile_name) botocore.exceptions.ProfileNotFound: The config profile (default) could not be found The problem looks like it might have been caused by this change: 7b38c7f ?diff=split?diff=split?diff=split%3Fdiff%3Dsplit <7b38c7f?diff=split?diff=split?diff=split%3Fdiff%3Dsplit> Is this a known issue, or this there a workaround for the problem I could apply? Thanks. — Reply to this email directly, view it on GitHub <#163>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADUYXVYESPMQBZQX2BWT3TXEFV2RANCNFSM6AAAAAAXTRIPQM> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

Matt Harvey · Answer 2 · Wed May 03 2023 07:22:21 GMT+0800 (China Standard Time)

There are no environment variables or profile setup on the EC2 instance for authentication. What I am guessing was happening previously was the session = botocore.session.get_session() gets credentials for the execution role associated with the EC2 instance (I may be wrong though).

Igor Okulist · Answer 3 · Wed May 03 2023 07:41:50 GMT+0800 (China Standard Time)

Alright I’ll recheck on ec2 instance. Which service are you accessing (ec2, lambda..)?On May 2, 2023, at 6:22 PM, Matt Harvey ***@***.***> wrote: There are no environment variables or profile setup on the EC2 instance for authentication. What I am guessing was happening previously was the session = botocore.session.get_session() gets credentials for the execution role associated with the EC2 instance (I may be wrong though). —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>

Matt Harvey · Answer 4 · Wed May 03 2023 08:11:03 GMT+0800 (China Standard Time)

Lambda via API Gateway endpoint. The API gateway endpoint is secured via IAM policy that restricts access based on AWS Organisation Units.

…

________________________________ From: Igor Okulist ***@***.***> Sent: Wednesday, 3 May 2023 11:42 am To: okigan/awscurl ***@***.***> Cc: Matt Harvey ***@***.***>; Author ***@***.***> Subject: Re: [okigan/awscurl] Credentials issue introduced in 0.27.0 when running on EC2 instance. (Issue #163) Alright I’ll recheck on ec2 instance. Which service are you accessing (ec2, lambda..)?On May 2, 2023, at 6:22 PM, Matt Harvey ***@***.***> wrote: There are no environment variables or profile setup on the EC2 instance for authentication. What I am guessing was happening previously was the session = botocore.session.get_session() gets credentials for the execution role associated with the EC2 instance (I may be wrong though). —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***> — Reply to this email directly, view it on GitHub<#163 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABHSEFSDHKDNVKBKOIMUKVDXEGLUTANCNFSM6AAAAAAXTRIPQM>. You are receiving this because you authored the thread.Message ID: ***@***.***>

Igor Okulist · Answer 5 · Thu May 04 2023 05:25:08 GMT+0800 (China Standard Time)

got pulled in with other items -- will review asap.

Igor Okulist · Answer 6 · Fri May 05 2023 23:26:39 GMT+0800 (China Standard Time)

Capturing nice way to repro in aws cloud shell:

[cloudshell-user@ip-10-4-127-13 ~]$ pip3 install -q awscurl==0.26
[cloudshell-user@ip-10-4-127-13 ~]$ awscurl --service s3 https://awscurl-sample-bucket.s3.amazonaws.com 
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>awscurl-sample-bucket</Name><Prefix></Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><IsTruncated>false</IsTruncated><Contents><Key>awscurl-sample-file.txt</Key><LastModified>2017-07-25T21:27:38.000Z</LastModified><ETag>&quot;d41d8cd98f00b204e9800998ecf8427e&quot;</ETag><Size>0</Size><Owner><ID>978427f78b00827efacb8fe2bd55ea30cbcb1d228cd3758972314cb67e763402</ID><DisplayName>okigan</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents></ListBucketResult>



[cloudshell-user@ip-10-4-127-13 ~]$ pip3 install -q awscurl==0.27
[cloudshell-user@ip-10-4-127-13 ~]$ awscurl --service s3 https://awscurl-sample-bucket.s3.amazonaws.com 
Traceback (most recent call last):
  File "/home/cloudshell-user/.local/bin/awscurl", line 8, in <module>
    sys.exit(main())
  File "/home/cloudshell-user/.local/lib/python3.7/site-packages/awscurl/awscurl.py", line 543, in main
    inner_main(sys.argv[1:])
  File "/home/cloudshell-user/.local/lib/python3.7/site-packages/awscurl/awscurl.py", line 508, in inner_main
    args.profile)
  File "/home/cloudshell-user/.local/lib/python3.7/site-packages/awscurl/awscurl.py", line 427, in load_aws_config
    cred = session.get_credentials()
  File "/usr/local/lib/python3.7/site-packages/botocore/session.py", line 508, in get_credentials
    'credential_provider'
  File "/usr/local/lib/python3.7/site-packages/botocore/session.py", line 1108, in get_component
    self._components[name] = factory()
  File "/usr/local/lib/python3.7/site-packages/botocore/session.py", line 187, in _create_credential_resolver
    self, region_name=self._last_client_region_used
  File "/usr/local/lib/python3.7/site-packages/botocore/credentials.py", line 74, in create_credential_resolver
    metadata_timeout = session.get_config_variable('metadata_service_timeout')
  File "/usr/local/lib/python3.7/site-packages/botocore/session.py", line 318, in get_config_variable
    logical_name
  File "/usr/local/lib/python3.7/site-packages/botocore/configprovider.py", line 426, in get_config_variable
    return provider.provide()
  File "/usr/local/lib/python3.7/site-packages/botocore/configprovider.py", line 628, in provide
    value = provider.provide()
  File "/usr/local/lib/python3.7/site-packages/botocore/configprovider.py", line 718, in provide
    scoped_config = self._session.get_scoped_config()
  File "/usr/local/lib/python3.7/site-packages/botocore/session.py", line 416, in get_scoped_config
    raise ProfileNotFound(profile=profile_name)
botocore.exceptions.ProfileNotFound: The config profile (default) could not be found
[cloudshell-user@ip-10-4-127-13 ~]$

Igor Okulist · Answer 7 · Sat May 06 2023 01:01:16 GMT+0800 (China Standard Time)

Output from the latest release:

[cloudshell-user@ip-10-2-12-122 ~]$ pip3 install -q awscurl==0.28
[cloudshell-user@ip-10-2-12-122 ~]$ awscurl --service s3 https://awscurl-sample-bucket.s3.amazonaws.com 
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>awscurl-sample-bucket</Name><Prefix></Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><IsTruncated>false</IsTruncated><Contents><Key>awscurl-sample-file.txt</Key><LastModified>2017-07-25T21:27:38.000Z</LastModified><ETag>&quot;d41d8cd98f00b204e9800998ecf8427e&quot;</ETag><Size>0</Size><Owner><ID>978427f78b00827efacb8fe2bd55ea30cbcb1d228cd3758972314cb67e763402</ID><DisplayName>okigan</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>awscurl-sample-file:.txt</Key><LastModified>2023-05-05T16:03:34.000Z</LastModified><ETag>&quot;d41d8cd98f00b204e9800998ecf8427e&quot;</ETag><Size>0</Size><Owner><ID>978427f78b00827efacb8fe2bd55ea30cbcb1d228cd3758972314cb67e763402</ID><DisplayName>okigan</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents></ListBucketResult>