publish v2 to PyPI
mattsb42-aws opened this issue · comments
It's great that there's a first-class supported installation mechanism that does not rely on environment specifics, but this is still a normal Python package, and for those customers who do already have an existing compatible environment the installation mechanism is a lot heavier-weight than using the existing Python distribution mechanisms.
Could we get v2 publishing to PyPI so that those customers who have those prerequisites can have a simpler and easier to maintain installation/updating/etc story?
Not publishing CLI v2 to PyPI has been a conscious decision and for the time being we have no plans to publish to PyPI. In the long term we want the fact that the AWS CLI is written in Python to be more of an implementation detail than a feature.
@joguSD Could you please elaborate on the reasoning behind this conscious decision? Perhaps link us to some dialog around this discussion?
I think that time may prove if that decision was more towards conscious or in-conscious one. I am really curious to see how this will evolve, I only hope I have enough popcorn 🍿.
On the other hand, other wheels coming from the same source did not get their notoriety without any reasons. boto, boto3 anyone?
At least pypi was 3rd party that guaranteed that a specific version is not altered once uploaded. I guess now we will have to keep saving checksums .
So pip install git+git://github.com/aws/aws-cli.git#2.0.0 then. :)
Sure, we all know that is impossible to force push code to github, not.
Or use a tarballs link for greater speed https://adamj.eu/tech/2019/03/11/pip-install-from-a-git-repository/
Just as a reminder, the AWS CLI V1 will continue to receive regular updates
that are published to PyPI and pip will continue to be a supported
installation mechanism for AWS CLI V1.
Here are some reasons why we dropped publishing to PyPI for AWS CLI V2, and are
not supporting pip as an official installation mechanism.
-
Installing with
piprequires Python knowledge. The AWS CLI isn't a tool
exclusive to Python developers and it's very easy to end up with an invalid
installation. Often, reported issues end up being due to a misconfiguredpip
install that's getting dependencies from the wrong place, etc. These issues
happen unexpectedly and are hard to debug which is frustrating for customers.
It's also possible to inadvertently break other packages in the same
environment. -
Depending on a C extension can mean that at
pip installtime a C compiler
and any other required libraries may need to be present for the install to
succeed. Because of this we've avoided depending on libraries that require C
extensions as having a compiler wasn't previously a requirement to install the
AWS CLI. We also do not feel comfortable adding a compiler as a requirement to
install the AWS CLI V2. For V1 this has limited our ability to implement some features
as there are no suitable Python native libraries that meet our requirements
(cryptography, etc). -
Supporting
pipas an installation mechanism requires supporting large
dependency ranges for the packages we depend on. While simple in concept, this
is difficult to execute on reliably as it's not feasible to test all
permutations of platform, Python version, and package version. When new
packages we depend on are published V1 customers installing viapipare often
broken. This leads us to have relatively aggressive version ranges and we end
up becoming a blocker for the Python community to upgrade packages.
See #4749 #4561 #3535 -
There are some things we have no control of when installing via
pip. For
example, we have no influence on theopensslversion or what TLS related
features are supported as those will depend on how Python was built. This makes
it difficult to ensure functionality of the AWS CLI. -
We cannot include or ship additional binaries (that may not even be written
in Python) as part of an AWS CLI install. In the future the AWS CLI can be
partially or even entirely rewritten in another language while maintaining
backwards compatibility.
Hopefully that helps to shed some light on why we decided not to publish the
AWS CLI V2 to PyPI.
Lastly, we're looking to continue to improve the installation process for AWS
CLI V2 by potentially integrating with homebrew and Linux package managers, and
providing docker images. We'll also be looking to include build instructions
for cases where the pre-built binaries aren't available or suitable for a
particular use case.
Those sound as decent and understandable reasons — except perhaps the #1. I'd avoid going into language comparison contests here — but it's hard to refute that Python is among the few must know languages for DevOps folks and the like.
Likely, I'm being ignorant in this judgement; but one can be forgiven for not "really knowing" even Bash — but not Python.
It's the language of humanity's first ever photo of a [shadow of] black hole, remember?
Besides: that's some really wierd reasoning, don't you think?
Some users have hard time withpip, while others (most?..) pip users are quite happy and silent.Yet others have hard time with prebuilt binaries, while others (most?..) binariy users are quite happy and silent.Therefore, we'd punish ALL pip users, including the happy&silent ones, and force them to go use binaries.
Edit: sorry, disregard this.
Lastly: pip install https://github.com/aws/aws-cli/archive/2.0.0.tar.gz fails with:
No matching distribution found for botocore==2.0.0dev4 (from awscli==2.0.0)
What's the route to go to install aws-cli v2 from source?
Not only aws-cli but also botocore(low-level API) is not published to PyPI for v2, so you need to install botocore beforehand.
For example, on a vanilla A1(ARM-base) Amazon Linux 2 instance, you can install AWS-CLI v2 as follows:
$ sudo yum install -y python3 python3-devel gcc libffi-devel openssl-devel
$ python3 -mpip install https://github.com/boto/botocore/archive/v2.tar.gz
$ python3 -mpip install https://github.com/aws/aws-cli/archive/v2.tar.gz
$ aws --version
aws-cli/2.0.0 Python/3.7.4 Linux/4.14.165-131.185.amzn2.aarch64 botocore/2.0.0dev4
What about providing an AppImage for Linux vs the zip archive with the installer? I created one using the contents from the dist dir and it seems to be working fine. Just throwing it out there as it may be worthwhile way to distribute the v2 release for Linux. I can share steps I used if there's any interest.
That like is like gold
No matching distribution found for botocore==2.0.0dev4 (from awscli==2.0.0)
Translates to "We made a release using unreleased dependencies." and "We didn't test if the release artifact can be installed".
Nobody mentioned is that binary code downloaded needs to signed on MacOS, so which means that the so called brew alternative install method will take considerable amount of effort/time.
Let's face it, v2 was released even without a tag being made of github, this should say a lot. AFAIK it could also be made from another repository, another license. The v2 was not presented as a preview and was directly published on https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html as a recommended release when the code repository, didn't had any tag, prerelease or not.
I would think twice before arguing that so many users are not able to use pip, or compile code, when basic software development practices like tagging are not followed.
Installing with pip requires Python knowledge...
I agree. Doing things in a domain requires domain knowledge.
Depending on a C extension can mean that at pip install time a C compiler
and any other required libraries may need to be present...
This is more domain knowledge, sure.
Supporting pip as an installation mechanism requires supporting...
It seems to me that this is your crux. Supporting PIP costs time and money and it sounds like it is generally painful. I suspect this is because, of the people that need support, they don't have the domain knowledge you'd mentioned.
There is a difference between "supporting PIP as the method of install" vs "uploading your package to PyPi". You can keep on delivering your curl/unzip/bash method of deployment as official and supported and also allow, for the people that know and want to, install v2 via pip as an "unsupported" option.
There are some things we have no control of when installing via pip...
You may not have control over it, but you can check for it and make it dependent on a specific version.
We cannot include or ship additional binaries...
Additional binaries... sounds like dependencies, which could be checked for during the build process.
The take-away I'm trying to make here is that supplying your packages via PIP isn't antithetical to your desire to limit support to your official installer. Would you please reconsider your aversion to pypi? Many developers would appreciate it.
There are some things we have no control of when installing via pip. For
example, we have no influence on the openssl version or what TLS related
features are supported as those will depend on how Python was built. This makes
it difficult to ensure functionality of the AWS CLI.
Don't you think that this is quite intentional?..
To a user of your package, that's an atrocious idea: that installing an API client tool gives control over openssl on my system to the tool author. I'm getting shivers just from thinking of it!
Constructively, there're two things you can do:
- outright fail pip build, if the given openssl isn't good enough;
- graceful degradation, where parts of functionality which depend on missing TLS features just get disabled.
We cannot include or ship additional binaries (that may not even be written
in Python) as part of an AWS CLI install. In the future the AWS CLI can be
partially or even entirely rewritten in another language while maintaining
backwards compatibility.
Why, what's wrong with setup(data_files=...) and/or MANIFEST.in, did you look into it?
Thousands of developers do include arbitrary files in their pip installs. I see no problem here.
Oh wait, actually I do. Binaries don't belong in source installs.
If I'd want to rewrite aws-cli in golang or something — I'd make it separate project, in a separate repo, and maintain its interface as "CLI compatible" rather than vaguely "backwards-compatible".
Don't ship binaries where source is expected. No-no. No no no no.
@joguSD Would it make sense to have separate issue for "install with a package manager" or "install with package manager X". Currently It's hard to automate the installation with the bundled installer (because you may not have a directory in your PATH that's suited for not-packaged binaries).
And it could be a good place to consolidate all the related issues like this one, #4926 #4842 #4660 #3553 #3627 #2501 and #727.
With v1 the suggestion was (probably rightly so) to use pip when needing a packaged/automated installation, with v2 this isn't an option anymore
@joguSD what about platforms you don't directly support but language packaging mechanisms (like PyPi) do? For instance, Cygwin?
Hi!
I do understand that AWS is trying to make it easier to install the AWS CLI for users who are not familiar with Python. It is regrettable that it is done at the expense of users who are familiar with Python, in a way that prevents them from integrating the AWS CLI within many environments.
For instance, many platforms will let you specify Python dependencies in a requirements.txt or similar file. A package published on PyPI becomes installable on these platforms without any extra or special work. A package installed through a script that requires root does not.
This is the case on Heroku, and it seems to be the case on AWS Lambda too. If I want to run a script using the AWS CLI on Lambda, it looks like I could do it with AWS CLI v1 (since the Lambda documentation explains how to package Python dependencies) but I don't know if it'd be possible with AWS CLI v2.
There are also many platforms that won't allow root access. A package published on PyPI can be installed as an unprivileged user, while the AWS CLI v2 installer requires sudo.
There are ways to have your cake and eat it too: check out Docker Compose installation instructions. Compose is available as a standalone executable for most platforms, and can also be installed with pip install.
I understand the desire to make Python an "implementation detail". But every software is written in some language, and that language has best practices around installation procedures. For instance, C code can leverage e.g. autoconf and automake, and every Linux distribution will have helpers to facilitate packaging software that uses autoconf and automake. Likewise, software written in Python can leverage PyPI.
Finally, if CLI installation is truly a huge point of friction, there are more drastic solutions as well. For instance, it could be ported to Go, to allow producing a standalone binary for every major platform. In 2012, I was working at Docker Inc. when we made the decision to rewrite our container engine from Python to Go, in order to facilitate its adoption by the largest number of people. I do understand that the AWS CLI is a complex piece of software that needs to interface with hundreds of services. But if a 15-person start-up (the headcount of Docker back then) can rewrite a container engine in Go, perhaps a company like AWS can manage to port a CLI as well? ;-)
If porting the CLI to Go seems to big a burden, I'd suggest to at least offer a PyPI installation process. It's fine if it's "not officially supported". At least don't make our job harder.
Thanks,
The installation instructions look like a alpha release note. However, the documentation suggests that v2 is a prod release. The only installation method that seems to be prod ready is Windows MSI (I have not tested).
Please provide options for installing using package managers such as pip etc ASAP.
Unfortunately a new core AWS feature (SSO) is a non-starter for us for the moment, until
- CLIv2 is available through some sort of versioned repo (pip, distro repos, anything) and not just an installer linked on an AWS documentation page
- Core SDK support for basic things like 'use SSO to authenticate' are available (this is purely directed at botocore).
Another problem with the current install process is that while there are instructions on how to update the package, there doesn't appear to be any documented way to programmatically check if an update is available.
I'm having to use the new aws sso login feature a lot, and since i'm working inside virtual environments, which use tools which have awscli as a dependency, I have to exit my virtual environment every time i need to use the sso login (hourly).
Please save me from this! please release to PyPI!!
For my own future reference 😄
pip install --user https://github.com/boto/botocore/archive/v2.zip https://github.com/aws/aws-cli/archive/v2.zip
## Not working anymore ...
pip install --user git+git://github.com/boto/botocore.git#v2 git+git://github.com/aws/aws-cli.git#v2
Another disadvantage with not publishing to pip, you making me have a second python interpreter installed, which might be fine for a dev machine, but in a docker image isnt good.
This is related to the issues installing on alpine linux in #4685 - if it just used the system python (as in v1) then would be no problem.
Installing with pip requires Python knowledge. The AWS CLI isn't a tool
exclusive to Python developers and it's very easy to end up with an invalid
installation. Often, reported issues end up being due to a misconfigured pip
install that's getting dependencies from the wrong place, etc. These issues
happen unexpectedly and are hard to debug which is frustrating for customers.
It's also possible to inadvertently break other packages in the same
environment.
Not sure I get your point here. Installing with pip does not mean pip has to be the only way to install aws cli.
e.g. There are already RPM and DEB packages of AWS CLI at the moment.
Depending on a C extension can mean that at pip install time a C compiler
and any other required libraries may need to be present for the install to
succeed. Because of this we've avoided depending on libraries that require C
extensions as having a compiler wasn't previously a requirement to install the
AWS CLI. We also do not feel comfortable adding a compiler as a requirement to
install the AWS CLI V2. For V1 this has limited our ability to implement some features
as there are no suitable Python native libraries that meet our requirements
(cryptography, etc).
Again, I do not get the point you are making here. Nobody is forcing AWS CLI to be distributed as a source python package. You can ship pre-built versions of AWS CLI as wheels to pypi, thus have no build-time requirement during installation.
There are some things we have no control of when installing via pip. For
example, we have no influence on the openssl version or what TLS related
features are supported as those will depend on how Python was built. This makes
it difficult to ensure functionality of the AWS CLI.
What are you implying here?
That you will now ship your own python interpreter and openssl inside your new AWS CLI "bundle"?
Considering your main point seem to be the support burden, I fear this decision will just change your support requests from "help me aws cli is broken because of my openssl version" to "please release a new bundle with an updated openssl because of CVE-42".
Will I have to independently update AWS CLI whenever there is a security fix in python/openssl ?
We cannot include or ship additional binaries (that may not even be written
in Python) as part of an AWS CLI install. In the future the AWS CLI can be
partially or even entirely rewritten in another language while maintaining
backwards compatibility.
Of course you can, multiple libraries on pypi do ship with prebuilt dependencies embedded.
You can either statically link your bindings with the depended library during wheel creation, or just directly embed the shared/dynamic library in your package and use that for your cffi calls, there's plenty of ways to do that.
To be honest, this move of getting out of pypi is really getting on my nerve.
The code quality of boto3 and AWS CLI is abysmal, and all I see here is a escape plan from that codebase.
I am pretty sure we will soon have an announcement along the lines of "Sorry, we screwed up in Python, we are now working on porting AWS CLI to Go, we hope it's going to be better <3"
Considering your main point seem to be the support burden, I fear this decision will just change your support requests from "help me aws cli is broken because of my openssl version" to "please release a new bundle with an updated openssl because of CVE-42".
This is especially bad, since there is not a great way to automate upgrading the AWS CLI now.
From all the reasoning I've read here against using pip as an installation vehicle (one reason being that not all folks are python savvy as you put it) I can only recommend using a language that compiles to binaries. AWS CLI rewritten in Golang would take care of most of these so called issues. Hence why installing anything from Hashicorp is relatively trivial...
Another reason for AWS to publish their own aws-cli v2 to PyPi: if you don't, someone else will!
Another reason for AWS to publish their own aws-cli v2 to PyPi: if you don't, someone else will!
yeah, but this one uses docker. So no thanks.
@mvanbaak this package was made mostly for CI, in my case - GitHub Actions. Just to avoid awscli dependency hell.I do not recommend it for local use.
@mvanbaak well, new version supports installation on Linux x86_64, Linux ARM and Mac OS with awsv2 --install. Docker image now is used as a fallback.
It would be great if someone could help me to test MacOS installer.
I made an attempt to package awscli v2 into Gentoo Linux came and conclusion that this is just not possible in any feasible way. The arguments dropped here are beyond ridiculous and the end result is that you've created a binary blob that can work only within it's own binary blob environment, let's get over those arguments.
Installing with pip requires Python knowledge. The AWS CLI isn't a tool
exclusive to Python developers and it's very easy to end up with an invalid
installation. Often, reported issues end up being due to a misconfigured pip
install that's getting dependencies from the wrong place, etc. These issues
happen unexpectedly and are hard to debug which is frustrating for customers.
It's also possible to inadvertently break other packages in the same
environment.
PYPI is the source of truth for Python projects, and such projects are packaged using PYPI as source, release to PYPI allows distribution to actually package what you have there in a way that will well integrate with whole Python ecosystem present, regardless of what it is.
Depending on a C extension can mean that at pip install time a C compiler
and any other required libraries may need to be present for the install to
succeed. Because of this we've avoided depending on libraries that require C
extensions as having a compiler wasn't previously a requirement to install the
AWS CLI. We also do not feel comfortable adding a compiler as a requirement to
install the AWS CLI V2. For V1 this has limited our ability to implement some features
as there are no suitable Python native libraries that meet our requirements
(cryptography, etc).
Then stop being afraid of adding such dependency, people who package your project will handle it, and those who install manually out of the distribution's scope can handle the need to install a compiller and handful of headers.
Supporting pip as an installation mechanism requires supporting large
dependency ranges for the packages we depend on. While simple in concept, this
is difficult to execute on reliably as it's not feasible to test all
permutations of platform, Python version, and package version. When new
packages we depend on are published V1 customers installing via pip are often
broken. This leads us to have relatively aggressive version ranges and we end
up becoming a blocker for the Python community to upgrade packages.
See #4749 #4561 #3535
That's a matter of interfaces. As long as interface remain unchanged, you should be good. You wouldn't need to support 'ranges' if your dependencies follow semantic versioning, then you can just depend on major.minor version knowing, that interface won't change in patch level version, if it however change, press your upstream to align to a common sense semantic versioning.
There are some things we have no control of when installing via pip. For
example, we have no influence on the openssl version or what TLS related
features are supported as those will depend on how Python was built. This makes
it difficult to ensure functionality of the AWS CLI.
Sure you can put a version of OpenSSL supported in README and make a check inside awscli if all the required interfaces are there, then print an error and exit if they're not. You know, like whole world has been doing the whole time.
We cannot include or ship additional binaries (that may not even be written
in Python) as part of an AWS CLI install. In the future the AWS CLI can be
partially or even entirely rewritten in another language while maintaining
backwards compatibility.
Which is not a problem as long as you release source code and GNU Make, CMake, Meson or whatever build system, then awscli can install Python's part, and have runtime dependency on non-pypi awscli-extras or whatever you would put in there, that is also to be packaged by distributions, this is not a new thing, all the time Python projects depends on non-python dependencies in runtime.
So the current situation is that awscli v2 is dependent on botocore v2, that is not released inside git repo (not tagged in any way) and you make remark that this is planned, because you have no desire to support installation out of your binary blob/Docker image, because... that would require puting a few checks and a README with dependencies. Your decision to ignore problem rather than address it made it literally impossible to integrate it into any distribution without bending over backwards and adding tons of hacks, just so your awscli can import unversioned botocore v2 that has no interface parity with v1, so the original botocore cannot be replaced with it.
So the current situation is that awscli v2 is dependent on botocore v2…made it literally impossible to integrate it into any distribution without bending over backwards and adding tons of hacks…
I did bend over backwards and add tons of hacks so, within my organization, we could have a wheel install of awscli 2.0. It goes like this: (Mine is in a Dockerfile, so pardon any careless filesystem usage).
A python script named botocore_is_old.py:
#!/usr/bin/env python3
"""To be run as a git bisect run command to determine if we have the version of botocore we want.
Thus, this is expected to be run in a botocore source directory."""
import botocore # This needs to be from the current directory, not another one on sys.path!
from packaging import version # You may need to pip install packaging, first
want = version.parse(open('WANT_BOTOCORE_VERSION').read())
have = version.parse(botocore.__version__)
print(f"Have: {have}; Want: {want}")
##
# Return 1 if the version we have is greater than or equal to the one we want,
# otherwise 0.
# git bisect will stop when it finds the first "bad" commit,
# which is actually the version we want.
raise SystemExit(int(have >= want))In the shell:
git clone -qb v2 https://github.com/aws/aws-cli aws-cli # Check out the branch v2
git clone -qb v2 https://github.com/boto/botocore botocore # Check out the branch v2
( cd aws-cli && awk -F= '/botocore/{print $3}' setup.cfg > ../botocore/WANT_BOTOCORE_VERSION )
( cd botocore &&
printf 'Working excessively hard to get botocore version ' \
&& cat WANT_BOTOCORE_VERSION \
&& if ./botocore_is_old.py; \
then git bisect start \
&& git bisect old "$(git merge-base origin/master HEAD)" \
&& git bisect new HEAD \
&& git bisect run ./botocore_is_old.py; \
fi )Having run that, the local botocore clone will be on some commit that has a version (according to packaging) that matches what the clone of awscli wants. You can then package up that botocore version and the aws-cli v2 however you want.
Obviously, this can fail quite easily, and probably will. I don't intend to maintain this hack in comments to this issue thread. But since I already did the work to this point, I offer it here in case anyone else is as stubborn as I am.
Right, the whole business with what's going on with botocore (not following symver or tags) makes living with the AWS position tenable. It's not an easy, straight forward way to get AWSCLIv2 installed from source b/c of the shenanigans in botocore.
Please add and support it on PyPI. I needs it in my requirements file to build cool stuff on AWS.
- Installing with pip requires Python knowledge. The AWS CLI isn't a tool
exclusive to Python developers and it's very easy to end up with an invalid
installation.
Bullshit, docker-compose is a python-based CLI tool, yet i did not even know of it until i once Ctrl-C'd in the middle of its initialisation phase (where it spat out a very nice KeyboardInterrupt exception in the middle of an import), many tools exist as python scripts like this, you can isolate installations and dependencies to be away from user-installed python libraries, or better yet, mesh nicely with existing dependencies installed and managed by the system's package manager.
- Depending on a C extension can mean that at pip install time a C compiler
and any other required libraries may need to be present for the install to
succeed.
Bullshit, wheels exist, and extensive CI tools to create them in as wide of a range exists (see cibuildwheel)
- Supporting pip as an installation mechanism requires supporting large
dependency ranges for the packages we depend on. While simple in concept, this
is difficult to execute on reliably as it's not feasible to test all
permutations of platform, Python version, and package version.
This is the price you pay for using python, python is constantly in flux in that fashion, all you can do is tread lightly in regards to dependency management, and have a hefty situation awareness about what's going on in the python world. (I see you guys still support 2.7, 3.4, and 3.5, what's up with that?)
- There are some things we have no control of when installing via pip. For
example, we have no influence on the openssl version or what TLS related
features are supported as those will depend on how Python was built. This makes
it difficult to ensure functionality of the AWS CLI.
Sorta hooking into #3 here as well: You can avoid this by not building your main CLI product in python, and instead statically building binaries which ensures complete control over any and all dependencies everywhere, for this i'd recommend using Rust or Golang. If dependency management is such a high item on the list, continuing to use python isn't the way to go, especially not with pip becoming more stricter about dependency management in the future
- We cannot include or ship additional binaries (that may not even be written
in Python) as part of an AWS CLI install.
Why not? If you're referring to the addition of binary blobs to installation packages (alongside source files), then I can get that. If you mean "we cannot ship binaries", then i'd be very interested why, for example, aforementioned wheels indeed wouldn't be a good distribution/targeting/shipping mechanism.
Hi, Is it possible to keep both the aws cli v1 and v2 installed on our linux remote servers? Please let me know if you have any idea. We can migrate to v2 completely but we dont want to change our development scripts. So we are just checking if there is any possibility of installing both the versions
@ddodla, as of right now, your option is to use Python virtual environments (I've left out all the messiness getting AWSCLIv2 installed from source with pip commands). AWSCLIv2 use to have a different name, which permitted this but it no longer does because it's "official" now.
Then you would "activate" which virtual environment you'd like to use (for your aws CLI invocations), e.g.
$ workon
ansible
aws_inventory
awscli
awscliv2
awssamcli
$ workon awscli
$ aws --version
aws-cli/1.18.160 Python/3.7.8 CYGWIN_NT-10.0-19041/3.1.7-340.x86_64 botocore/1.19.0
$ deactivate
$ workon awscliv2
$ aws --version
aws-cli/2.0.56 Python/3.7.8 CYGWIN_NT-10.0-19041/3.1.7-340.x86_64 source/x86_64
Then you would "activate"
This is a very useful answer, but I still feel the need to point out you don't need to activate a virtualenv to use a single binary from it. If you invoke the python script from the virtualenv's path, directly, it will usually run as in the virtualenv.
path/to/aws-v1-venv/bin/aws s3 lsand
path/to/aws-v2-venv/bin/aws s3 lsshould both work. Also, workon is part of virtualenvwrapper, not virtualenv itself.
Just want to mention this, to nuke the 'possible dependency issues when installing awscli' argument: one can use pipx to install it.
Just pipx install awscli (or, hopefully, a PyPI release of awscli v2) and Bob's your uncle.
- No dependency clashes because
awscligets installed in its own virtual environment, along with its dependencies. - No command clashes with other installed packages because
pipxby default only installs binaries and scripts provided by the installed (main) application, not from any of its dependencies. - If
awscliv2 requires a particular Python version, just install that withpyenvand add--python python<version>to thepipx installcommand. - Easy upgrades: just run
pipx upgrade awscli.
All this can be packed up into a nice curl | bash script for the dumb and/or lazy. And yes, pipx runs on Linux, Windows and macOS.
The result of doing just that on my own system just a minute ago:
$ pipx install awscli
installed package awscli 1.18.166, Python 3.9.0
These apps are now globally available
- aws
- aws.cmd
- aws_bash_completer
- aws_completer
- aws_zsh_completer.sh
done! ✨ 🌟 ✨
$ pipx inject awscli awscli-plugin-eucalyptus
injected package awscli-plugin-eucalyptus into venv awscli
done! ✨ 🌟 ✨
$ aws --version
aws-cli/1.18.166 Python/3.9.0 Linux/5.8.16_1 botocore/1.19.6
$ pipx runpip awscli list
Package Version
------------------------ --------
awscli 1.18.165
awscli-plugin-eucalyptus 0.2
botocore 1.19.5
colorama 0.4.3
docutils 0.15.2
jmespath 0.10.0
pip 20.2.4
pyasn1 0.4.8
python-dateutil 2.8.1
PyYAML 5.3.1
rsa 4.5
s3transfer 0.3.3
setuptools 50.3.2
six 1.15.0
urllib3 1.25.11
wheel 0.35.1
(Yes, I injected awscli-plugin-eucalyptus package into awscli virtual environment (yet another trick which pipx has up its sleeve) for good measure ;) )
Folks, I just have created a single binary AWS CLI v2 using AppImage. Now the installation is as easy as below:
curl -L https://github.com/simnalamburt/awscliv2.appimage/releases/latest/download/aws-x86_64.AppImage -o aws
chmod +x awsVisit my repository and take a look!
Folks, I just have created a single binary AWS CLI v2 using AppImage.
That's great, @simnalamburt, but I don't see how it helps folks that want to pip install awscli v2, using python ecosystem tools like pip and virtualenv. As far as I understand, that's what this ticket is about.
Some people in this thread are looking for the concise installation mechanism, so I thought it’s worth a comment.
Folks, I just have created a single binary AWS CLI v2 using AppImage. Now the installation is as easy as below:
curl -L https://github.com/simnalamburt/awscliv2.appimage/releases/latest/download/aws-x86_64.AppImage -o aws chmod +x awsVisit my repository and take a look!
That is not a solution.
It is frankly insulting to package this simple AWS client into an appimage, a format meant for user programs, while some/most people would rather integrate this into their workflow without messing around with packaging and format issues.
How hard is it to just bite the bullet and throw a py3-none wheel on pypi? It does the same thing and it's easily accessible by any and all existing python packaging tools, an appimage like this, a diversion, is just a disappointment.
Edit: oh fuck sorry, i thought you were a AWS employee, that criticism isn't warranted in any way then, my apologies
My point is simply: how hard can it be for AWS to do python setup.py bdist_wheel and throw the output on pypi?
The way I see it is that someone though that maintaining packages and versions is 'hard' and they're 'too agile' to care about it so they decided to open a path to bundle whatever abomination they want, in any language and binary form and ship it as such blob, sounds good, right?
This is a common pattern. First you have team that tries their best to do things right, like release and version the core bindings (botocore), release and version SDK (boto3), but over time this become a blocker for the team. A lot of effort is required to release things and track interfaces, they are also limited to Python and whatever they can bundle with CPython, they want more, they want to be able to drop whatever they want into package, and they want to release fast. So what the team does? they ignore the common sense and create Docker image, AppImage or a big .zip with all the Python runtime, all the dependencies, because this way they're not dependent on end-user runtime, they do not need to care about interfaces backward compatibility, they can do breaking changes and as long as the big release tarball have all of what's needed they don't care.
What awscli v2 needs is a release management, interface tracking and integration with end-user systems. If they broke all the interfaces in 'botocore v2' they can just fork it into botocore-2, fork awscli into awscli-2 and maintain and version it separately, as long as the modules name contain the suffix, one can have the 'old' and 'new agile' botocore installed at the same time, no one is hurt this way, and you can go back to a proper release process.
[...] they ignore the common sense and create Docker image, AppImage or a big .zip with all the Python runtime [...]
I understand the sentiment, but notice: somebody has to deal with the packaging issues (tracking interfaces, versioning ranges, compatibility with dependencies, etc) one way or another — otherwise, the software decays, falls apart over time. Plopping out a zip file (Docker image, AppImage) as your release with everything bundled @ single supported version — is just pushing out integration responsibilities onto your users. Somebody will need that SDK to work with newer Python; somebody will need to get it working with patched openssl compiled for a weird cpu architecture, weird country crypto standards & custom HSM support; almost everyone will need aws-cli v2 to be installable via pip just like v1 is.
That's the observation that this "common pattern" ignores: package authors are the ones best positioned to apply maintenance efforts — not package users.
I think it should be obvious by now: developers aren't users, we don't desire simplicity over correctness and explicitness, we desire determinism and integration, and simplicity flowing from that.
pip is a wonderful tool, and it has an amazing ecosystem of packages around it, seriously, why not use that?
And if alternative solutions have to be found because pip forces you to have a correct and sound (as of 20.3, with their new resolver) dependency structure, then that's a failure of your software process, not of the platforms you're publishing to.
My point is simply: how hard can it be for AWS to do
python setup.py bdist_wheeland throw the output on pypi?
Yep, that's what I am doing now (building a wheel) but they aren't tagging 2.x.x. releases in botocore. You have to read the log to find the commit in the v2 branch which is the version depended on by AWSCLI 2.x.x. :-(, so YMMV... Usually, it's not much effort but I have grabbed botocore updates that were further ahead of the AWSCLI 2.x.x dep on botocore, forcing me to do a little reading...
they aren't tagging 2.x.x. releases in botocore
why not?
So I can install SAM CLI from PyPi with pip3, but I have to install additional dependencies (unzip) in my Docker image, just so that I can install the AWS CLI v2.
Simple Install
Simple is better.
pip3 install aws-sam-cli # One line, done
Complex Install
apt-get update
apt-get upgrade --yes
apt-get install unzip --yes
# Let me go dig up the documentation for AWS CLI v2 ....
# Oh wait, I need curl, too
apt-get install curl --yes
# What version am I installing from this URL? I have no idea.
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
# Oh wait, sudo doesn't exist in my Docker image
./aws/install
# Great, now I have two separate Python runtimes I have to worry about
root@de6f00100054:/# python3 --version
Python 3.8.5
root@de6f00100054:/# aws --version
aws-cli/2.1.21 Python/3.7.3 Linux/4.19.128-microsoft-standard exe/x86_64.ubuntu.20 prompt/off
@pcgeek86 awesome 🤣
Oh people, for Pete's sake! Python has had built-in support for handling ZIP files since like... version 1.6 already?? At the very least Python 2.7 has it. Don't developers read documentation anymore?
Well, I'm in a good mood, so here's a link (for Python 3, but Python 2 has been enjoying retirement for some time now).
Here is a little snippet you can paste into a file unzip.py to extract the contents of a ZIP:
import zipfile
with zipfile.ZipFile("file.zip","r") as zip_ref:
zip_ref.extractall("targetdir")You can then run it using python unzip.py.
I'll leave changing it so it accepts a filename and perhaps a directory to extract to as command line arguments as an excercise to the reader. We can't all be lazy now, can we?
Enjoy this great marvel of Python's 'batteries included' versatility.
(I'll let you in on a little secret, only for advanced wizards! You can load zipfile as a module on the command line, and all kinds of sweet magic start to happen. Hint: try python -m zipfile -h)
You can load zipfile as a module on the command line
This is a great tip, but nevertheless misses the point of this issue. pip install should be a supported way to install awscli2.
You can load zipfile as a module on the command line
This is a great tip, but nevertheless misses the point of this issue. pip > install should be a supported way to install awscli2.
Of course you are right, my comment was meant tongue in cheek.
No project should force its users to download and install an entire separate Python environment built from who knows what sources just to run a single tool.
There are shops out there that have very strict rules about what software is allowed to run on devops' machines, usually the need to vet the source code from which that software is built, and no random binary blobs downloaded from the internet, even if they are hatched by Amazon, are ever going to fit that bill.
If you want to sell your services you better bend over backwards to make those tools readily available in a professional manner. Meaning packages in all Linux distros ... skip pypi if you want but things like https://bugs.gentoo.org/767268#c1 are shocking results of a really poor deployment channel for that essential component. sudo funny-script-without-signature-or-checksum ...
Removing the path of least resistance (installing via pip) just makes me not want to upgrade. I'm sure a lot of other people will ignore v2 until v1 is EOL and we're forced to install a python cli tool in a non native way.
I'm sure a lot of other people will ignore v2 until v1 is EOL
I'm sorry if this is bitter feedback for some, but having learned that AWS staff has difficulties with basic packaging and versioning so much that they want to make Python an implementation detail of aws-cli — I'm reluctant to use even v1. Who knows what other nonsense they decided to put into there; I don't care enough to review the 70kSLoC myself.
$ wget https://awscli.amazonaws.com/AWSCLIV2-2.1.26.pkg
$ open AWSCLIV2-2.1.26.pkg
$ aws --version
aws-cli/2.1.26 Python/3.7.4 Darwin/20.3.0 exe/x86_64 prompt/offMeanwhile, a pip-installed CLIv1 runs fine on Python 3.9.
Now, I think therein lies a tale... 😄
I'm clearly late to this conversation... but just to highlight a major issue again: package managers are there for a reason. Throwing away package management altogether is such an incredible antipattern. There is so much that package managers do in the ops space.
Even in the environment of the dev workstation, where there's a human to control everything, what used to be
[package manager] install [package]
is now
- hunt for webpage with link
- run a command to pull the package down
- find a temp dir to extract it to
- scan the install script because it's a custom script doing a custom thing and who knows what it might do - hope you're familiar with shell!
- run the install
- test it
That's the recommended "happy path" with no edge cases. And then repeat that all again when there's an update... if you remember to check for one. This is not the right method for less-skilled admins, plus rather than "doing the work once" by the developers, you're making every admin duplicate the work.
If not pip because you want to go language agnostic, then why not rpm for Amazon Linux and maybe deb because it's the other super-popular package format? Both of these are designed for dependency management and deal with upgrades fluidly, and you don't have to hunt for a link in a webpage for packages in them.
I don't have the venv or docker problems that other users have but I'll keep using v1 anyway, because updating to get new features is a one-liner, rather than a manual zipfile install method from last century (no joke there - rpm's from '97, apt's from '98, cpan '95...)
The reason this is an anti-pattern for development is that a different version of botocore may be installed for other reasons, and so we Python developers who use virtual environments simply cannot use awscli v2 without using the workarounds above, because it would conflict with the packages used by other tools.
This works against proper development practices.
Some community work-arounds, such as pypi package awscliv2, work for me only at home because within my federal agency, we do not yet have access to Docker or WSL, and python virtual environments are my way to manage all dependency isolation.
@danizen i would recommend keeping working with venvs, because they're so widely used, and so more manageable than complete system environments, just a note.
we do not yet have access to Docker or WSL, and python virtual environments are my way to manage all dependency isolation.
Also, I dont want to install and run docker just for the awscli. It's not that I cant install it, it's just that I dont use docker, and having to go that route just for a cli tool ... yeah, that's a NOPE
Here you have a very well done paper on why the route you've choose with awscli v2 is wrong -- https://blogs.gentoo.org/mgorny/2021/02/19/the-modern-packagers-security-nightmare/
Unbelievable. And the worst part is - it's Amazons official. I wouldn't be surprised if it was Alibaba cloud or something, but it's Amazon. Really ? Wow, just wow.
This issue isn't going to progress anything, it's not getting closed nor resolved, but kept open to blackhole discussion, nothing will come from this, and nobody can force amazon to do anything about it. So i suggest finding other ways to forward this issue, other than leaving behind a comment here saying your support.
I've been watching this issue thread for a while now, and I haven't seen a member respond to the latest concerns and calls for python best-practice here, other than suggesting an entirely un-ergonomic way of deployment, not even willing to meet the developers half-way, dismissing releasing on pypi in favour of a vendored appimage blob.
Earlier here it's already been said that amazon considers using python in general as an implementation detail to how the aws cli is run, fine, fair, but then it acts as if it's above any choice it makes in that regard, e.g. not answering to what the community of python expects of their standardised way of publishing a package for a 30-year-old programming language, on a 20-year-old publication site, for a cloud service that has then only existed for 15 years, and then act as if their way is the highway.
Well then, @aws, fucking commit, make your own https://pypi.aws instance, dont just sit around and dabble with this absolute bullshit. You love developers so much? You love your customers so much? Prove it then, your platform has been a hot piece of overpriced garbage for a decade and a half now, so you cant change my mind, but you might change others.
Just boycott awscli v2. Don't use it.
If anyone ever wonders why — explain. Hopefully you can be understood.
I do see the problem with installing in a virtual environment from AWS perspective - there can be as many installations of AWS CLI as there are virtual environments, and together with user knowledge and the issue of including other capabilities maybe not written in Python and controlling the version of SSL (LibreSSL probably on their horizon), it makes sense.
However, I have programs that use boto3 to do things similar to aws sso login, and they rely on boto3 and therefore botocore.
Bottom-lines:
- If it cannot be published to PyPI then you need to assure that when I invoke the command-line aws from within a virtual environment, it uses its own version of botocore. I will be testing this and reporting back with V2 on Windows and then on Mac.
- It has to be published in a package format where we can programmatically determine updates. This is part of Software Supply Chain management for developers & DevOps, but also part of IT Asset Management, Users should be able to programmatically determine whether there is a new update available.
- It needs to be installable without administrative privileges on Windows, due to the many users who use AWS CLI who do not have administrative privileges to their own computer. Git for Windows does this - it is not that hard.
If these three requirements can be met by AWS, I see no problem with the lack of PyPI. Making an image available addresses the first two requirements, but the 3rd requirement is not met because many Windows users still are not allowed to use WSL.
It has to be published in a package format where we can programmatically determine updates. This is part of Software Supply Chain management for developers & DevOps, but also part of IT Asset Management, Users should be able to programmatically determine whether there is a new update available.
This is this biggest thing for me. I'd actually prefer having it in an apt repo to PyPi, but the current only official deployment mechanism is... not good. I also don't love that an entire python runtime is bundled with it. Or that if you upgrade the aws-cli, it leaves the old version lying around.
Well, we are at 2021 and rely on a zip file and bash script with couple cps and lns to install python libraries, how cool is that ?
Seriously, I now need to figure out how to install it with a SaltStack on all my ec2 instances, making it idempotent and upgradable.
Thanks for thinking about making it easier for DevOps.
It's so easy to upgrade awscli v1.
pipx upgrade awscli
Done. No root privileges required. It's what all the paying AWS customers want.
Instead we get this 20 step process which requires a dedicated installation page to explain (instead of a couple lines of the README file) which discourages upgrading. This probably creates more of a risk than anything else honestly.
I too am probably going to want to write a Salt state to manage this, but how many AWS customers are going to bother with that? Shouldn't AWS be taking on the inconveniences themselves instead of trying to push them onto paying customers?
Installing with pip requires Python knowledge.
Then continue to provide the already existing installation methods too.
Depending on a C extension can mean that at pip install time a C compiler and any other required libraries may need to be present for the install to succeed.
If you're already supplying another installation method, what does it matter? It's the user's decision.
Supporting pip as an installation mechanism requires supporting large dependency ranges for the packages we depend on.... See #4749 #4561 #3535
Just say you won't support anyone who isn't using a dedicated virtualenv environment for awscli. That would prevent you from dealing with all of those issues.
There are some things we have no control of when installing via pip. For example, we have no influence on the openssl version or what TLS related features are supported as those will depend on how Python was built.
Yes but it's not that hard to check and fail installation if your requirements are not met, with the appropriate error message. I think most users wouldn't have a problem, unless you decide to rely on some bleeding edge library feature (which wouldn't be a great idea anyway).
We cannot include or ship additional binaries (that may not even be written in Python) as part of an AWS CLI install.
Then have awscli initiate a check for whatever dependencies there might be (if ever that becomes a thing) at runtime where required. It's not a big deal.
But even if it was, this isn't your typical free software project where people can only hope, beg and perhaps offer a donation to help persuade a change in how things are done. In this case, you are getting paid to write this library because we are all paying for the AWS service. Hence I would expect AWS developers to take on some of the inconvenience to satisfy customers.
Well that's my thoughts on the matter. I'm a bit frustrated because I went to use aws ecr get-login as per the docs it wouldn't work, and I had to investigate why pipx wasn't upgrading my awscli setup, until I stumbled upon this huge issue.
@joguSD any response yet? Or are y'all explicitly ignoring this issue, even when it's attracting a lot of attention (even today) because you're ignoring one of the oldest language's best practices? Do you all really think you're so above that to not bother with these "silly practices"?
I re-read your first message, if you're not going to use python for awscli, commit, don't fuck around and say "oh but we might not do that", it's been a year, people are still annoyed at this not being on pypi, put it on pypi.
Ironically I can see a few of the Amazon leadership principles having been neglected and even broken with how they treat this issue. For one I don't see "Customer Obsession" being respected with this approach...
Hi everyone. First off, thank you all for the frank feedback about this decision, particularly specific feedback from some on this issue about why this matters to you. While @joguSD has already detailed why the team decided to not support pip, the continuing feedback is very important to us and we do use these anecdotes in future planning.
However, I would like to ask that we try to keep this conversation civil and constructive. Just as a friendly reminder, contributions to this repository, including issue comments, are governed by the Amazon Open Source Code of Conduct.
For our part, I can promise that we're reconsidering this decision internally based on the feedback we've received here. We want to consider the implications of a such a change (see @joguSD's comment linked above), so that will take some time. Expect an update from the team on that front by the end of next week.
The aws cli is not without bugs. There have been times I've had to step my way through a command in a debugger like PyCharm to find the issue. For this, it is very useful to be able to install via pip or pipx.
Even if I clone this project and check out v2 branch I cannot install with pip install -e . because it's depending on some non-published botocore==2.0.0dev97
Even if I clone this project and check out
v2branch I cannot install withpip install -e .because it's depending on some non-publishedbotocore==2.0.0dev97
I am guessing that is by design and intentional? They aren't tagging v2.x.x releases in botocore. To find the "2.0.0dev97 release" (ahem) of botocore you must read the commit log messages from the v2 branch. Then check out out the commit. Then build a wheel like python setup.py bdist_wheel. Then install or upgrade via pip (with the corresponding awscli v2 wheel). YMMV...
Hi all - I’m the manager of the AWS CLI team, and I wanted to address the conversation that’s been going on here, as well as some broader issues. I appreciate everyone who’s taken the time to comment. Every comment on every issue in this repository is read by multiple members of the team, and we value every anecdote and data point about how our product works or doesn’t work for you. One of our goals for the rest of 2021 is to do a better job of making that communication bi-directional and to more directly engage with the community on longer-term plans and roadmaps. I’ll discuss some of our plans for that, but first I want to address the issue at hand.
We do not plan to publish the AWS CLI project to PyPI at this time. I appreciate that this will not be a popular decision among people following this issue. In Q2, we plan to begin work on a source bundle that is well documented, easily ingestible by package managers, and that can be used to build the CLI for platforms other than those we support out of the box. This source distribution will include botocore along with the CLI. Our hope is that providing an officially-supported source distribution will alleviate many of the frustrations that you have experienced with our current approach. We understand that we need to make our product easily available for those who can’t or don’t want to use one of our pre-packaged binaries or container images. Once we’ve released and validated the source bundle, we’ll open an issue here to prioritize distribution channels that will provide the most value and ensure that maintainers have what they need from us.
Starting in April, we’re going to begin posting monthly roadmap updates as issues on our GitHub repository. These will lay out what we’re actively working on and what’s on the horizon. Our backlog includes a mix of priorities, some driven by the CLI team and the community, some which align with larger AWS goals. Our focus will be on sharing the priorities emerging from our team, which will include Github issues we are planning to work on. We will iterate on the exact format and approach for these updates as we determine what works best, but they will be at least monthly for the rest of 2021.
I’m going to lock this issue now, and we will be closing any new iterations of it as duplicates. I would ask that everyone please keep in mind the Code of Conduct https://aws.github.io/code-of-conduct when using GitHub to communicate with the team.