Support AWS SSO
iainelder opened this issue · comments
Does awscurl already support AWS SSO?
I tried just now and I seem to be missing something.
I'm using the version installed today directly from Github.
I have a config file with a profile that uses AWS SSO to get temporary credentials.
https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html
[profile example]
sso_start_url = https://d-1111111111.awsapps.com/start
sso_region = eu-west-1
sso_account_id = 111111111111
sso_role_name = AdministratorAccess
region = eu-west-1
output = jsonThe profile works with AWS CLI v2.
$ aws sts get-caller-identity --profile example
{
"UserId": "AROAAAAAAAAAAAEXAMPLE:example",
"Account": "111111111111",
"Arn": "arn:aws:sts::111111111111:assumed-role/AWSReservedSSO_AdministratorAccess_1111111111111111/example"
}
But when I try to use it with awscurl I get a message about the token being expired.
$ export AWS_PROFILE=example
$ awscurl --region=eu-west-1 --service=s3 http://example.s3.eu-west-1.amazonaws.com/
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>ExpiredToken</Code><Message>The provided token has expired.</Message><Token-0>IQoJ[...]96F0=</Token-0><RequestId>D[...]W</RequestId><HostId>GeFN[...]kfUTs=</HostId></Error>
Traceback (most recent call last):
File "/home/isme/.local/bin/awscurl", line 8, in <module>
sys.exit(main())
File "/home/isme/.local/pipx/venvs/awscurl/lib/python3.8/site-packages/awscurl/awscurl.py", line 500, in main
inner_main(sys.argv[1:])
File "/home/isme/.local/pipx/venvs/awscurl/lib/python3.8/site-packages/awscurl/awscurl.py", line 494, in inner_main
response.raise_for_status()
File "/home/isme/.local/pipx/venvs/awscurl/lib/python3.8/site-packages/requests/models.py", line 943, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http://example.s3.eu-west-1.amazonaws.com/
It looks like it's supported in boto3 1.14.0/botocore 1.17.
https://github.com/boto/botocore/blob/develop/CHANGELOG.rst#1170
https://github.com/boto/boto3/blob/develop/CHANGELOG.rst#1140
I don't see a version specified in your requirements.txt. Which version are you using?
Hm, now I'm confused.
I installed awscurl in a virtualenv with pipx.
It appears the botocore package isn't actually installed in the virtualenv.
Am I missing something?
I'm not sure if this is the right place to look for the botocore verison; I'm not very sure about how Python dependencies work in this context.
$ pipx runpip awscurl freeze
awscurl @ git+https://github.com/okigan/awscurl@f60961b4c6725a21bc4532a04d36c7f756636054
certifi==2020.12.5
cffi==1.14.5
chardet==4.0.0
ConfigArgParse==1.4
configparser==5.0.2
cryptography==3.4.7
idna==2.10
pkg-resources==0.0.0
pycparser==2.20
pyOpenSSL==20.0.1
requests==2.25.1
six==1.15.0
urllib3==1.26.4
Have not used pipx (so can't comment too much about that) -- since awscurl does not specifically depends on boto you'd need to specify / install boto libarary
According to requirements.txt awscurl seems to depend not on boto but on botocore.
Line 4 in f60961b
I don't understand why botocore isn't an installed dependency when it appears in requirements.txt.
Indeed botocore is optionally loaded in the code that parses the credentials.
Lines 379 to 392 in f8f7664
That part comes after some custom parsing of the AWS config file.
Why is there custom parsing of the config file before using botocore?
I thought (but I might be wrong!) that botocore was capable of parsing the credentials file already.
If botocore were used, it might be possible to support AWS SSO profiles without any custom code (but I haven't tested that).
botocore would be used if you have it installed and don't have any other credentials already set.
Do you have environment variables set with old access_key, secret_key, security_token?
I was quite sure of it. I will try to produce a repro for you when I have a moment.
I hadn't forgotten about this issue. I just took a long time to find some time :-)
I've got an explanation for the behavior that I reported and a solution to my request. But first I want to explain something that was confusing me before.
I don't understand why botocore isn't an installed dependency when it appears in requirements.txt.
Because the installation process appears to ignore requirements.txt. Instead it follows setup.py.
In setup.py botocore is declared as an optional dependency.
Lines 20 to 28 in f60961b
To install optional dependencies you need to change the installation command.
To test variations of the installation commands, I will use the same command as before with the --verbose flag for some debugging output that makes it easier to see what's happening inside the script. The complete test command looks like this:
awscurl \
--verbose \
--profile example \
--region=eu-west-1 \
--service=s3 \
http://example.s3.eu-west-1.amazonaws.com/Originally I had installed awscurl with just the required dependencies like this:
pipx install awscurlThe test command's verbose output contains lines like these:
'loading botocore package'
'botocore package could not be loaded'
So botocore could not be loaded because the package was not installed as a dependency.
Now I know to install awscurl with the optional awslibs feature like this:
pipx install awscurl[awslibs]
The test commands verbose output now contains lines like these:
'loading botocore package'
Implicitly we see that botocore was successfully loaded. This is because the package was installed as a dependency.
Have not used pipx (so can't comment too much about that)
Pipx is for avoiding dependency version conflicts for Python-based applications. Each package is installed to its own virtual environment. It's my first choice for installing awscurl and many other tools written in Python.
But when I try to use it with awscurl I get a message about the token being expired.
Do you have environment variables set with old access_key, secret_key, security_token?
Yes, I surely did.
When I use tools that don't understand AWS SSO profiles (Terraform pre-0.14.6 is another example), I use a tool called aws2-wrap to bridge the gap. It generates temporary credentials from the SSO profile and writes them to the shared credentials file. As a command line tool, it allows me to use a single script to automate the credential refresh process for all SSO profiles.
The AWS CLI v2 ignores the shared credentials file for SSO profiles. It uses its own cache for temporary credentials. Those credentials have a different expiry time. That explains why I was seeing expired credentials for awscurl but not for AWS CLI v2.
My initial feature request was a little unclear because at the time I had forgotten about that detail.
Now that I understand more about how awscurl's dependency management and how it loads credentials, I'm ready to provide a PR to add support for AWS SSO.
The good news is, as awscurl already depends on botocore, it's a small change to make.
As a workaround, you can use Leapp.
it generates AWS access, secret key, and an access token directly from AWS SSO credentials, making this automatically support AWS sso:
https://docs.leapp.cloud/use-cases/aws_sso/
But when I try to use it with awscurl I get a message about the token being expired.
Do you have environment variables set with old access_key, secret_key, security_token?
Yes, I surely did.
When I use tools that don't understand AWS SSO profiles (Terraform pre-0.14.6 is another example), I use a tool called aws2-wrap to bridge the gap. It generates temporary credentials from the SSO profile and writes them to the shared credentials file. As a command line tool, it allows me to use a single script to automate the credential refresh process for all SSO profiles.
The AWS CLI v2 ignores the shared credentials file for SSO profiles. It uses its own cache for temporary credentials. Those credentials have a different expiry time. That explains why I was seeing expired credentials for awscurl but not for AWS CLI v2.
My initial feature request was a little unclear because at the time I had forgotten about that detail.
Now that I understand more about how awscurl's dependency management and how it loads credentials, I'm ready to provide a PR to add support for AWS SSO.
The good news is, as awscurl already depends on botocore, it's a small change to make.
Where are we on this ? I would like to jump in if not started already.
I ended up using AWS Sigv4 Proxy. It's pretty simple and straight forward. All you need is docker container running in the background. Your credentials directory mounted inside the container (though I am sure binary can be used independently). Enviroment variable pointing which profile to use. (This is one time task).
Going forward make a normal curl request to localhost and make sure to manually add host header in the request. That's all.
Where are we on this ?
I moved on from awscurl to use other tools. I haven't found a use case for awscurl since then.
Thanks for the tip about AWS Sigv4 Proxy.
I don't use aws2-wrap any more either. Now I recommend the AWS SSO CLI. Its exec command allows you to execute other commands in a subshell with session environment variables.
+1 on this
A good workaround is to go to the AWS SSO start page and grab a temporary key and secret from there
For any mac-users who have installed awscurl with brew; you can add botocore to its virtualenv like this: $(brew --prefix awscurl)/libexec/bin/python -m pip install botocore
@iainelder thank you so much for this detailed comment #114 (comment) !!!
I did spent a bit of time debugging myself, but this oneliner saved me a bunch of extra time
pipx install 'awscurl[awslibs]'
Originally 'botocore' was selected to be optional to reduce the number dependencies installed.
Overtime, I have not heard complains of installing too many dependencies, but users do run into issues of not having 'botocore', so on next release I am inclined to make 'botocore' to be included by default...