botocore does not ignore credential_process when that configuration value is blank (empty string)

[Matthew X. Economou]

Describe the bug

The AWS CLI documentation on changing configuration settings using commands says:

To remove a setting, use an empty string as the value, or manually delete the setting in your config and credentials files in a text editor.

However, if one sets credential_process to the empty string, botocore neither removes the configuration variable from the configuration file nor ignores its value. Instead, subsequent invocations of the AWS CLI or botocore library result in an IndexError when botocore.credentials.ProcessProvider._retrieve_credentials_using() passes an empty list derived from the empty string value to subprocess.Popen() at botocore/credentials.py line 1019.

Expected Behavior

botocore (and by extension the AWS CLI) should act as if the credential_process configuration variable is unset when it has an empty string value as described in the AWS CLI documentation linked above.

$ rm ~/.aws/*
$ aws configure set credential_process "" --profile test
$ cat ~/.aws/config
[profile test]
credential_process = 
$ aws s3api list-buckets --profile test

Unable to locate credentials. You can configure credentials by running "aws configure".

Current Behavior

One cannot remove the credential_process setting without editing the configuration file.

$ rm ~/.aws/*
$ aws configure set credential_process "" --profile test
$ cat ~/.aws/config
[profile test]
credential_process = 
$ aws s3api list-buckets --profile test

list index out of range

Reproduction Steps

I've trimmed the output to focus on the expected behavior (NoCredentialsError in botocore.auth.add_auth() when no credentials are set) versus the current behavior (IndexError in subprocess._execute_child() when credentials_process is set to the empty string).

$ rm -rf ~/.aws
$ python3.11 -m venv .venv
$ source .venv/bin/activate
$ pip install awscli
$ aws configure set credential_process "" --profile test
$ python
>>> import botocore.session
>>> session = botocore.session.get_session()
>>> client = session.create_client('ec2', region_name='us-west-2')
>>> for reservation in client.describe_instances()['Reservations']:
...     for instance in reservation['Instances']:
...         print(instance['InstanceId'])
... 
botocore.exceptions.NoCredentialsError: Unable to locate credentials
>>> exit()
$ env AWS_DEFAULT_PROFILE=test python
>>> import botocore.session
>>> session = botocore.session.get_session()
IndexError: list index out of range

Possible Solution

The simplest fix may be to treat the value of credential_process as a boolean when deciding whether to use it in botocore.credentials.ProcessProvider.load() at botocore/credentials.py line 997. The current behavior of botocore as described above is obviously wrong.

While one might also modify awscli.customizations.configure.writer.ConfigFileWriter._update_section_contents() to delete configuration variables with empty string values, some configuration variables have different behavior when they do not exist versus when they are set to the empty string, e.g., cli_pager. I think the required logic to handle these cases would require too much effort to maintain for this approach to be worthwhile. Or one might define an aws configure subcommand that deletes configuration variables, but that just shifts the maintenance burden of when to delete a configuration variable versus when to set it to the empty string from the ConfigFileWriter code to the AWS CLI or botocore documentation.

Additional Information/Context

I didn't realize that AWS CLI version 2 uses its own version of botocore (cf. aws/aws-cli#6494). However, the problematic behavior is identical.

SDK version used

botocore 1.34.18 and AWS CLI 1.32.18, botocore 2.0.0dev155 and AWS CLI 2.15.10

Environment details (OS name and version, etc.)

Python 3.11.7 installed via MacPorts on macOS 12.7.1 (x86_64)

[Ryan F.]

Hi @rdctmeconomou, thanks for this detailed bug report. I was able to reproduce this behavior. After talking with the team, we came to the conclusion that the documentation is pretty misleading here. Setting a value to an empty string to remove a configuration option does not work for every option, and as you pointed out, causes issues for some. Given this information, we intend to remove that line from the documentation. At this time, our intention is to create an aws configure unset command in the future as the official way to remove configuration options. However, in the meantime, the best way to remove an option would be to manually delete it from the config file in a text editor. Please let me know if you have any follow-up questions.

[github-actions]

Greetings! It looks like this issue hasn’t been active in longer than five days. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.