`--ssl-no-verify` / `--ca-bundle` / `AWS_CA_BUNDLE` is not honored for `AWS_CONTAINER_CREDENTIALS_FULL_URI`
synfinatic opened this issue · comments
Describe the bug
I've created my own service to implement the AWS_CONTAINER_CREDENTIALS_FULL_URI protocol to securely manage AWS Identity Center IAM Role credentials.
The problem is that my service must use a self-signed certificate since it listens on https://localhost:4144. In a perfect world, I would be able to add the self-signed cert (in PEM format) to the embedded AWS truststore, but this doesn't seem to work.
Expected Behavior
SSL validation passes and the AWS Boto3 SDK makes the necessary HTTPS request to fetch the credentials stored at the HTTP endpoint at AWS_CONTAINER_CREDENTIALS_FULL_URI.
Current Behavior
SSL validation does not pass. AWS Boto3 SDK is unable to fetch the necessary credentials and errors out.
Reproduction Steps
Testing with AWS_CA_BUNDLE:
$ export AWS_CONTAINER_CREDENTIALS_FULL_URI=https://localhost:4144/
$ export AWS_CA_BUNDLE=`pwd`/localhost.crt
$ aws sts get-caller-identity
SSL validation failed for https://localhost:4144/ [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1006)--ca-bundle doesn't work either:
$ unset AWS_CA_BUNDLE
$ aws --ca-bundle=`pwd`/localhost.crt sts get-caller-identity
SSL validation failed for https://localhost:4144/ [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1006)Trying to disable SSL validation all together doesn't seem to work either:
$ aws --ssl-no-verify sts get-caller-identity
SSL validation failed for https://localhost:4144/ [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1006)Possible Solution
Honor the --ca-bundle, AWS_CA_BUNDLE and --no-ssl-verify flags/environment variables when connecting to the endpoint defined at AWS_CONTAINER_CREDENTIALS_FULL_URI
Additional Information/Context
No response
SDK version used
aws-cli/2.17.5 Python/3.11.9 Darwin/23.5.0 source/arm64
Environment details (OS name and version, etc.)
macOS 14.5
So I thought the issue might be the use of a self-signed certificate, but creating a server certificate signed by my internal CA and then passing that CA into aws by both methods does not solve the issue.
$ openssl s_client -CAfile CA.crt localhost:4144
Connecting to 127.0.0.1
CONNECTED(00000005)
Can't use SSL_get_servername
depth=1 CN=SynFindotNetCA, ST=California, L=San Jose, O=Syn Fin dot Net
verify return:1
depth=0 CN=localhost, C=US, ST=California, L=San Jose, O=Syn Fin dot Net, OU=aws-sso-cli
verify return:1
---
Certificate chain
0 s:CN=localhost, C=US, ST=California, L=San Jose, O=Syn Fin dot Net, OU=aws-sso-cli
i:CN=SynFindotNetCA, ST=California, L=San Jose, O=Syn Fin dot Net
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Jul 6 15:30:28 2024 GMT; NotAfter: Jul 4 15:30:28 2034 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFnzCCA4egAwIBAgIBDDANBgkqhkiG9w0BAQsFADBbMRcwFQYDVQQDEw5TeW5G
aW5kb3ROZXRDQTETMBEGA1UECBMKQ2FsaWZvcm5pYTERMA8GA1UEBxMIU2FuIEpv
c2UxGDAWBgNVBAoTD1N5biBGaW4gZG90IE5ldDAeFw0yNDA3MDYxNTMwMjhaFw0z
NDA3MDQxNTMwMjhaMHkxEjAQBgNVBAMTCWxvY2FsaG9zdDELMAkGA1UEBhMCVVMx
EzARBgNVBAgTCkNhbGlmb3JuaWExETAPBgNVBAcTCFNhbiBKb3NlMRgwFgYDVQQK
Ew9TeW4gRmluIGRvdCBOZXQxFDASBgNVBAsTC2F3cy1zc28tY2xpMIIBIjANBgkq
hkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAuA8jrATxYTcmRanYIzcGj4HWDU7SgWiM
b7CseCgt8H2FS9Yki/7P+5UaUKnRAfGNOqdl+upez/oP+cO3X+eYz1Yfn854BMCU
e8yiGk/wQAyzLx6AKGib+d3t1NHdAmKpQRqfBkekmVcI/6ba2E0IpyVT7A+gYZ19
ayfJrIwbnjc/U/DdnhXcrZ5jfnLDHKYenDGfnLon9axrtVYycb8+BQ00/RTv+nbS
Gi+Ny48e/B1HDlAK93ZMV1kYMV6QWTtiQy6dz314zN7FvPpJQfHA90CNasrFGoyD
xKcSOoXwi5YYT/mb/oAERkt1DTnjCPujgOiZCN21gQ5KAbOcWFQCzwIDAQABo4IB
TjCCAUowCQYDVR0TBAIwADARBglghkgBhvhCAQEEBAMCBkAwCwYDVR0PBAQDAgWg
MDMGCWCGSAGG+EIBDQQmFiRPcGVuU1NMIEdlbmVyYXRlZCBTZXJ2ZXIgQ2VydGlm
aWNhdGUwHQYDVR0OBBYEFBZuaKHF2jYOkNWT0UTifBKUWPYhMIGDBgNVHSMEfDB6
gBTXvRSzQDmHEiU4NXhDGwGVnuohZaFfpF0wWzEXMBUGA1UEAxMOU3luRmluZG90
TmV0Q0ExEzARBgNVBAgTCkNhbGlmb3JuaWExETAPBgNVBAcTCFNhbiBKb3NlMRgw
FgYDVQQKEw9TeW4gRmluIGRvdCBOZXSCAQAwJwYDVR0lBCAwHgYIKwYBBQUHAwEG
CCsGAQUFBwMCBggrBgEFBQgCAjAaBgNVHREEEzARgglsb2NhbGhvc3SHBH8AAAEw
DQYJKoZIhvcNAQELBQADggIBAIOrYd42F92dvclp8XS+k/cNOB3z02UL5R5PhEK6
PnHVkPNhlFouFjadREBIaUEl/WQaZ3wR2jmO7grKdlRBRE2pNkJMH33PhWC2iO8t
l24I5QeQTmVsmrADdDNil/Rb6iS3ItVtNwPYBZ0RHY3IKHXSZBlDAUefH+rm5YnS
8DV56vjx6tf4Jc9iJxD/NRCl9TUBiMiMsVXEAPLpYFqnLvpPT2PJMTnYM9YdilIu
+kI+1kaOurrznnGAhWNJCm/wfH9DRH2ar5ajuzLJKkWvPd7No3YEBdO3Y4odPHfP
M6ktyjQXl65LaJgatlLkIPY5I6DoKtXBVHqUhP2XTOC4xO4SUMT7l8+NqQAWTMI6
twRGZHqID0qCatCNu6ezeUzMKW0R9asVckwCJcZPzlU0DMbUsE7z+8IBL5fGZhDo
ZkqceZe1MSey511QtJX5y+/1ofggZsQkZtW0IdxU67Kn5VFcLMP4r/xc1DAyqvl0
MaMsA+GdaZh2NV9Gt8ANvB1j1WrOocHW6HIsCOU9rXwHHn6bYEFfMEQa4akwsfxj
D9wQcT5Go5RXcvPhERIc7xw2Ojsz6L0rg2XZA6tgxe+q24bv2+4BwYcjNj6VVfIW
kj3uTGyXf/MZ1VQD7zihKCr3jNza87LidXBnksawwT27eQHqQ+9NtPJTv1Oeh9M+
iPkd
-----END CERTIFICATE-----
subject=CN=localhost, C=US, ST=California, L=San Jose, O=Syn Fin dot Net, OU=aws-sso-cli
issuer=CN=SynFindotNetCA, ST=California, L=San Jose, O=Syn Fin dot Net
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1983 bytes and written 363 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 2048 bit
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
Protocol : TLSv1.3
Cipher : TLS_AES_128_GCM_SHA256
Session-ID: A74AC870CE2D4AD3DE7E3A273F8EC43EC9E8CDE1318CE19C71F8C5B242825DD0
Session-ID-ctx:
Resumption PSK: B4DA1DAB2D2D495B6FF489C6D58509750FE94A32B4BC9FD9AFD24681402C47C2
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 604800 (seconds)
TLS session ticket:
0000 - 6e d3 12 d5 03 e8 61 e5-0d 62 25 a2 41 09 e7 21 n.....a..b%.A..!
0010 - 67 77 87 cc ad 80 92 6a-b1 e8 eb fd ec d4 a1 12 gw.....j........
0020 - b8 6b fe 00 7e ca 1b 0b-8a 4d 4e 38 dc 6a 09 b4 .k..~....MN8.j..
0030 - 1e 7c 6a f0 73 a0 2a d0-7f d2 09 1f 02 5b 38 d5 .|j.s.*......[8.
0040 - d4 88 2f 86 89 ce 41 58-81 a0 88 10 3c 23 fc ea ../...AX....<#..
0050 - f2 d4 a3 b0 9d 01 f1 79-0f b7 2f ab 74 f3 2b 8b .......y../.t.+.
0060 - 1b b8 6b 75 45 03 c8 ba-79 ..kuE...y
Start Time: 1720280214
Timeout : 7200 (sec)
Verify return code: 0 (ok)
Extended master secret: no
Max Early Data: 0
---
read R BLOCKoutput from aws-cli:
$ aws sts get-caller-identity --ca-bundle CA.crt --no-verify-ssl --debug
2024-07-06 08:40:11,120 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.17.5 Python/3.11.9 Darwin/23.5.0 source/arm64
2024-07-06 08:40:11,120 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['sts', 'get-caller-identity', '--ca-bundle', 'CA.crt', '--no-verify-ssl', '--debug']
2024-07-06 08:40:11,127 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_s3 at 0x103a544a0>
2024-07-06 08:40:11,127 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_ddb at 0x10386ade0>
2024-07-06 08:40:11,127 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.configure.configure.ConfigureCommand'>>
2024-07-06 08:40:11,127 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x1034a7560>
2024-07-06 08:40:11,127 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x1034c8c20>
2024-07-06 08:40:11,127 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function alias_opsworks_cm at 0x103a6c540>
2024-07-06 08:40:11,127 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_history_commands at 0x1038c2200>
2024-07-06 08:40:11,127 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.devcommands.CLIDevCommand'>>
2024-07-06 08:40:11,127 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_waiters at 0x103a574c0>
2024-07-06 08:40:11,127 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x103b2ded0>>
2024-07-06 08:40:11,127 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/data/cli.json
2024-07-06 08:40:11,128 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_types at 0x103982160>
2024-07-06 08:40:11,128 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function no_sign_request at 0x103982480>
2024-07-06 08:40:11,128 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_verify_ssl at 0x1039823e0>
2024-07-06 08:40:11,128 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_read_timeout at 0x1039825c0>
2024-07-06 08:40:11,128 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_connect_timeout at 0x103982520>
2024-07-06 08:40:11,128 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <built-in method update of dict object at 0x103b27280>
2024-07-06 08:40:11,128 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.17.5 Python/3.11.9 Darwin/23.5.0 source/arm64
2024-07-06 08:40:11,128 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['sts', 'get-caller-identity', '--ca-bundle', 'CA.crt', '--no-verify-ssl', '--debug']
2024-07-06 08:40:11,128 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_timestamp_parser at 0x103a54ea0>
2024-07-06 08:40:11,128 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function register_uri_param_handler at 0x1032c3e20>
2024-07-06 08:40:11,128 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_binary_formatter at 0x103af0fe0>
2024-07-06 08:40:11,128 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function no_pager_handler at 0x103163600>
2024-07-06 08:40:11,128 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x1032cbce0>
2024-07-06 08:40:11,129 - MainThread - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/
2024-07-06 08:40:11,133 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function attach_history_handler at 0x1038c19e0>
2024-07-06 08:40:11,133 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_json_file_cache at 0x10384ae80>
2024-07-06 08:40:11,139 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/data/sts/2011-06-15/service-2.json
2024-07-06 08:40:11,140 - MainThread - botocore.hooks - DEBUG - Event building-command-table.sts: calling handler <function add_waiters at 0x103a574c0>
2024-07-06 08:40:11,146 - MainThread - botocore.hooks - DEBUG - Event building-command-table.sts: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x103b2ded0>>
2024-07-06 08:40:11,146 - MainThread - awscli.clidriver - DEBUG - OrderedDict()
2024-07-06 08:40:11,146 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.sts.get-caller-identity: calling handler <function add_streaming_output_arg at 0x103a554e0>
2024-07-06 08:40:11,146 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.sts.get-caller-identity: calling handler <function add_cli_input_json at 0x1032eca40>
2024-07-06 08:40:11,146 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.sts.get-caller-identity: calling handler <function add_cli_input_yaml at 0x1032ecae0>
2024-07-06 08:40:11,146 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.sts.get-caller-identity: calling handler <function unify_paging_params at 0x10386b560>
2024-07-06 08:40:11,152 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/data/sts/2011-06-15/paginators-1.json
2024-07-06 08:40:11,152 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.sts.get-caller-identity: calling handler <function add_generate_skeleton at 0x103980860>
2024-07-06 08:40:11,152 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.sts.get-caller-identity: calling handler <bound method OverrideRequiredArgsArgument.override_required_args of <awscli.customizations.cliinput.CliInputJSONArgument object at 0x103babed0>>
2024-07-06 08:40:11,152 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.sts.get-caller-identity: calling handler <bound method OverrideRequiredArgsArgument.override_required_args of <awscli.customizations.cliinput.CliInputYAMLArgument object at 0x1031ec2d0>>
2024-07-06 08:40:11,152 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.sts.get-caller-identity: calling handler <bound method GenerateCliSkeletonArgument.override_required_args of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x103ba4250>>
2024-07-06 08:40:11,152 - MainThread - botocore.hooks - DEBUG - Event building-command-table.sts_get-caller-identity: calling handler <function add_waiters at 0x103a574c0>
2024-07-06 08:40:11,152 - MainThread - botocore.hooks - DEBUG - Event building-command-table.sts_get-caller-identity: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x103b2ded0>>
2024-07-06 08:40:11,152 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sts.get-caller-identity.cli-input-json: calling handler <awscli.paramfile.URIArgumentHandler object at 0x100f2e150>
2024-07-06 08:40:11,152 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sts.get-caller-identity.cli-input-yaml: calling handler <awscli.paramfile.URIArgumentHandler object at 0x100f2e150>
2024-07-06 08:40:11,152 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sts.get-caller-identity.generate-cli-skeleton: calling handler <awscli.paramfile.URIArgumentHandler object at 0x100f2e150>
2024-07-06 08:40:11,153 - MainThread - botocore.hooks - DEBUG - Event calling-command.sts.get-caller-identity: calling handler <bound method CliInputArgument.add_to_call_parameters of <awscli.customizations.cliinput.CliInputJSONArgument object at 0x103babed0>>
2024-07-06 08:40:11,153 - MainThread - botocore.hooks - DEBUG - Event calling-command.sts.get-caller-identity: calling handler <bound method CliInputArgument.add_to_call_parameters of <awscli.customizations.cliinput.CliInputYAMLArgument object at 0x1031ec2d0>>
2024-07-06 08:40:11,153 - MainThread - botocore.hooks - DEBUG - Event calling-command.sts.get-caller-identity: calling handler <bound method GenerateCliSkeletonArgument.generate_skeleton of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x103ba4250>>
2024-07-06 08:40:11,153 - MainThread - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/
2024-07-06 08:40:11,153 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTP connection (1): 169.254.169.254:80
2024-07-06 08:40:11,154 - MainThread - botocore.utils - DEBUG - Caught retryable HTTP exception while making metadata service request to http://169.254.169.254/latest/api/token: Could not connect to the endpoint URL: "http://169.254.169.254/latest/api/token"
Traceback (most recent call last):
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/connection.py", line 174, in _new_conn
conn = connection.create_connection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/util/connection.py", line 95, in create_connection
raise err
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/util/connection.py", line 85, in create_connection
sock.connect(sa)
OSError: [Errno 64] Host is down
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/httpsession.py", line 448, in send
urllib_response = conn.urlopen(
^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/connectionpool.py", line 801, in urlopen
retries = retries.increment(
^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/util/retry.py", line 527, in increment
raise six.reraise(type(error), error, _stacktrace)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/packages/six.py", line 770, in reraise
raise value
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/connectionpool.py", line 715, in urlopen
httplib_response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/connectionpool.py", line 416, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/connection.py", line 244, in request
super(HTTPConnection, self).request(method, url, body=body, headers=headers)
File "/opt/homebrew/Cellar/python@3.11/3.11.9/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1303, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/awsrequest.py", line 94, in _send_request
rval = super(AWSConnection, self)._send_request(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/python@3.11/3.11.9/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1349, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/opt/homebrew/Cellar/python@3.11/3.11.9/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1298, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/awsrequest.py", line 122, in _send_output
self.send(msg)
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/awsrequest.py", line 206, in send
return super(AWSConnection, self).send(str)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/python@3.11/3.11.9/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 996, in send
self.connect()
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/connection.py", line 205, in connect
conn = self._new_conn()
^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/connection.py", line 186, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <botocore.awsrequest.AWSHTTPConnection object at 0x103bbff50>: Failed to establish a new connection: [Errno 64] Host is down
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/utils.py", line 387, in _fetch_metadata_token
response = self._session.send(request.prepare())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/httpsession.py", line 477, in send
raise EndpointConnectionError(endpoint_url=request.url, error=e)
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "http://169.254.169.254/latest/api/token"
2024-07-06 08:40:11,160 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTP connection (2): 169.254.169.254:80
2024-07-06 08:40:11,160 - MainThread - botocore.utils - DEBUG - Caught retryable HTTP exception while making metadata service request to http://169.254.169.254/latest/meta-data/placement/availability-zone/: Could not connect to the endpoint URL: "http://169.254.169.254/latest/meta-data/placement/availability-zone/"
Traceback (most recent call last):
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/connection.py", line 174, in _new_conn
conn = connection.create_connection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/util/connection.py", line 95, in create_connection
raise err
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/util/connection.py", line 85, in create_connection
sock.connect(sa)
OSError: [Errno 64] Host is down
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/httpsession.py", line 448, in send
urllib_response = conn.urlopen(
^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/connectionpool.py", line 801, in urlopen
retries = retries.increment(
^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/util/retry.py", line 527, in increment
raise six.reraise(type(error), error, _stacktrace)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/packages/six.py", line 770, in reraise
raise value
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/connectionpool.py", line 715, in urlopen
httplib_response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/connectionpool.py", line 416, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/connection.py", line 244, in request
super(HTTPConnection, self).request(method, url, body=body, headers=headers)
File "/opt/homebrew/Cellar/python@3.11/3.11.9/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1303, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/awsrequest.py", line 94, in _send_request
rval = super(AWSConnection, self)._send_request(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/python@3.11/3.11.9/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1349, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/opt/homebrew/Cellar/python@3.11/3.11.9/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1298, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/awsrequest.py", line 122, in _send_output
self.send(msg)
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/awsrequest.py", line 206, in send
return super(AWSConnection, self).send(str)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/python@3.11/3.11.9/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 996, in send
self.connect()
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/connection.py", line 205, in connect
conn = self._new_conn()
^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/connection.py", line 186, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <botocore.awsrequest.AWSHTTPConnection object at 0x103c6a2d0>: Failed to establish a new connection: [Errno 64] Host is down
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/utils.py", line 437, in _get_request
response = self._session.send(request.prepare())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/httpsession.py", line 477, in send
raise EndpointConnectionError(endpoint_url=request.url, error=e)
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "http://169.254.169.254/latest/meta-data/placement/availability-zone/"
2024-07-06 08:40:11,161 - MainThread - awscli.utils - DEBUG - Max number of attempts exceeded (1) when attempting to retrieve data from metadata service.
2024-07-06 08:40:11,161 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2024-07-06 08:40:11,161 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2024-07-06 08:40:11,161 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role-with-web-identity
2024-07-06 08:40:11,161 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: sso
2024-07-06 08:40:11,161 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file
2024-07-06 08:40:11,161 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: custom-process
2024-07-06 08:40:11,161 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: config-file
2024-07-06 08:40:11,164 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: ec2-credentials-file
2024-07-06 08:40:11,164 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: boto-config
2024-07-06 08:40:11,164 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: container-role
2024-07-06 08:40:11,164 - MainThread - botocore.httpsession - DEBUG - Certificate path: /opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/cacert.pem
2024-07-06 08:40:11,164 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): localhost:4144
2024-07-06 08:40:11,176 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/connectionpool.py", line 715, in urlopen
httplib_response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/connectionpool.py", line 404, in _make_request
self._validate_conn(conn)
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/connectionpool.py", line 1060, in _validate_conn
conn.connect()
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/connection.py", line 419, in connect
self.sock = ssl_wrap_socket(
^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
ssl_sock = _ssl_wrap_socket_impl(
^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/python@3.11/3.11.9/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ssl.py", line 517, in wrap_socket
return self.sslsocket_class._create(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/python@3.11/3.11.9/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ssl.py", line 1104, in _create
self.do_handshake()
File "/opt/homebrew/Cellar/python@3.11/3.11.9/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ssl.py", line 1382, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/httpsession.py", line 448, in send
urllib_response = conn.urlopen(
^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/connectionpool.py", line 801, in urlopen
retries = retries.increment(
^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/util/retry.py", line 527, in increment
raise six.reraise(type(error), error, _stacktrace)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/packages/six.py", line 769, in reraise
raise value.with_traceback(tb)
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/connectionpool.py", line 715, in urlopen
httplib_response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/connectionpool.py", line 404, in _make_request
self._validate_conn(conn)
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/connectionpool.py", line 1060, in _validate_conn
conn.connect()
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/connection.py", line 419, in connect
self.sock = ssl_wrap_socket(
^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
ssl_sock = _ssl_wrap_socket_impl(
^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/python@3.11/3.11.9/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ssl.py", line 517, in wrap_socket
return self.sslsocket_class._create(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/python@3.11/3.11.9/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ssl.py", line 1104, in _create
self.do_handshake()
File "/opt/homebrew/Cellar/python@3.11/3.11.9/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ssl.py", line 1382, in do_handshake
self._sslobj.do_handshake()
urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/clidriver.py", line 499, in main
return command_table[parsed_args.command](remaining, parsed_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/clidriver.py", line 634, in __call__
return command_table[parsed_args.operation](remaining, parsed_globals)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/clidriver.py", line 837, in __call__
return self._operation_caller.invoke(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/clidriver.py", line 959, in invoke
client = self._session.create_client(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/session.py", line 850, in create_client
credentials = self.get_credentials()
^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/session.py", line 456, in get_credentials
'credential_provider').load_credentials()
^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/credentials.py", line 1974, in load_credentials
creds = provider.load()
^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/credentials.py", line 1829, in load
return self._retrieve_or_fail()
^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/credentials.py", line 1838, in _retrieve_or_fail
creds = fetcher()
^^^^^^^^^
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/credentials.py", line 1867, in fetch_creds
response = self._fetcher.retrieve_full_uri(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/utils.py", line 2796, in retrieve_full_uri
return self._retrieve_credentials(full_url, headers)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/utils.py", line 2841, in _retrieve_credentials
return self._get_response(
^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/utils.py", line 2855, in _get_response
response = self._session.send(request.prepare())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/awscli/2.17.5/libexec/lib/python3.11/site-packages/awscli/botocore/httpsession.py", line 475, in send
raise SSLError(endpoint_url=request.url, error=e)
botocore.exceptions.SSLError: SSL validation failed for https://localhost:4144/ [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)
SSL validation failed for https://localhost:4144/ [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)So I tried using the Go SDK v2 and looks like it's having the same issue, so this seems more of a general AWS SDK issue, and not specific to the Python/boto3 SDK:
package main
import (
"context"
"fmt"
"os"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/service/sts"
)
func main() {
ca, err := os.Open("./CA.crt")
if err != nil {
panic(fmt.Sprintf("Unable to open CA.crt: %v", err))
}
cfg, err := config.LoadDefaultConfig(context.TODO(), config.WithCustomCABundle(ca))
if err != nil {
panic(err)
}
client := sts.NewFromConfig(cfg)
output, err := client.GetCallerIdentity(context.TODO(), nil)
if err != nil {
panic(err)
}
fmt.Printf("identity: %v\n", output)
}$ ./aws-test
panic: operation error STS: GetCallerIdentity, get identity: get credentials: failed to refresh cached credentials, failed to
load credentials, exceeded maximum number of attempts, 3, request send failed, Get "https://localhost:4144/": tls: failed
to verify certificate: x509: “localhost” certificate is not trusted
goroutine 1 [running]:
main.main()
/Users/aturner/go/src/github.com/synfinatic/aws-sso-cli/cmd/aws-test/main.go:25 +0x194
Thanks for reaching out. Here is where AWS_CONTAINER_CREDENTIALS_FULL_URI is documented in the SDKs & Tools Reference Guide: https://docs.aws.amazon.com/sdkref/latest/guide/feature-container-credentials.html. Also referenced there:
If you use Amazon ECS, we recommend you use a task IAM Role for improved credential isolation, authorization, and auditability. When configured, Amazon ECS sets the
AWS_CONTAINER_CREDENTIALS_RELATIVE_URIenvironment variable that the SDKs and tools use to obtain credentials.
Have you tried that approach for ECS?
Can you also confirm that your certificate is valid, and no proxy/network configurations are causing the error here? This troubleshooting guide on SSL validation errors highlights common causes: https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-troubleshooting.html#tshoot-certificate-verify-failed
Otherwise since this issue applies to behavior across SDKs (as environment variables such as that one are used across AWS SDKs), could you create an issue for this in our cross-SDK repository sharing your findings up to this point?
Thanks. Opened aws/aws-sdk#774
This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.