Add support for urllib3 2.2.1
brodycj opened this issue · comments
Describe the bug
Underlying botocore requires outdated urllib3 version:
- 1.x in case of Python pre-3.10
- < 2.1 in case of Python 3.10(+)
background from: #2926
I do totally understand the motivation for these outdated urllib3 version dependencies. IMHO I think it should be up to whoever uses boto3 to specify whether or not to use an outdated version of urllib3.
Incidentally I am using this for an application that may run on Amazon Linux, which may need the older urllib3 dependency, would have preferred to make this choice myself.
I am taking liberty to file this issue for boto3, no objections if maintainers want to transfer this to botocore.
I am also taking liberty to leave out some info that I think should not matter - please post a loud holler if any further info is needed. Thanks in advance!
Expected Behavior
I think it should be possible to install boto3 and another package such as requests with recently updated version of urllib3 such as urllib3 == 2.2.1.
Current Behavior
Combinaiton of boto3 == 1.34.46 & urllib3 == 2.2.1 leads to pip installation error.
Reproduction Steps
see above - please post a loud holler if any further info is needed
Possible Solution
No response
Additional Information/Context
No response
SDK version used
Python 3.9 / 3.11; boto3 v1.34.46
Environment details (OS name and version, etc.)
Ubuntu 22.04.3 LTS 64-bit (with some customized updates) on Intel core i7
God please yes!
Collecting urllib3<1.27,>=1.25.4 (from botocore<1.35.0,>=1.34.48->boto3<2,>=1.9.253->watchtower)
Using cached urllib3-1.26.18-py2.py3-none-any.whl.metadata (48 kB)
I keep having to do this
pip install watchtower --no-dependencie
pip install boto3 --no-dependencie
pip install botocore --no-dependencie
Hi everyone, just to reiterate what was discussed in the original issue since it was left out of the description. Botocore cannot support urllib3 2.0 on Python <3.10, this is covered in the urllib3 docs as well. There are no plans currently to change that as it would result in broad impact for AWS customers until they manually pin themselves.
For Python 3.10+, urllib3 2.2.0 had breaking changes that were impacting botocore until the most recent release on Saturday (Feb 17). We're currently reviewing the latest version to confirm there are no other subtle breakages and will update this ticket once we have a clear path to moving the pin. Thanks for your patience!
One more thing is that the application I am working on also uses requests, which uses urllib3 in turn. I continue to wish that botocore could leave this up to the higher-level application. This was part of my original reqeust.
P.S. Yes I do totally understand the need to keep the functionality stable, avoiding breaking changes from underlying modules such as urllib3. But there will be users who would rather avoid using outdated versions, especially if they may affect any other libraries. I guess this is a situation we cannot avoid given the design of Python & PIP installations.
Just to clarify, Requests has no requirements to update to urllib3 2.0, it's compatible with both the 1.x and 2.x branches. urllib3 1.26.x is still actively maintained and is receiving continued releases for the foreseeable future.
But yes, you're right, a lot of the defensiveness in our pins is due to how Python's packaging infrastructure works. Without the pins we'd have had ~4-5 large scale breakages for boto3 which is the most downloaded python package. In the rare occassions those do happen, it's felt across a large portion of the python ecosystem. We have to weigh safety over total flexibility since there are breakages with a major version bump in dependencies.
Can't you just vendor urllib3?
Can't you just vendor urllib3?
The short answer is no. We already vendored urllib3 for years (botocore>=1.0.0,<=1.12.253) before ultimately removing it from botocore and similarly, was removed from Requests. The primary downsides are size and longevity of releases. Once there's a vulnerability in urllib3, every version of botocore with an older vendored copy becomes either an active security risk or wasted space on PyPI. Keeping the two projects separated allows for more flexibility in choices, as well as not requiring us to make a release for customers to patch their systems.
The other major issue is the size of botocore. While already large, adding vendored dependencies further increases overall size and create problems when packaged for Lambda. The downsides of vendoring outweigh the majority of the benefits which we already have empirical data on. While it's unfortunate we may not have same day parity with new releases, the trade off is stability at the expense of a moderate delay.
Why not have botocore 1.x for ancient Python versions and botocore 2.x for all the modern cool kids who stick with active stable releases and actually maintain their code base?
Thanks for the feedback, @robd003. To make sure we're on the same page, all versions of urllib3 we support are actively maintained, stable releases. urllib3 1.x is still receiving updates and we're currently ~3 weeks behind on the first stable release of urllib3 for 2.x beyond what we currently support.
Major versioning botocore would make this packaging problem considerably worse. Anyone using the AWSCLI, PynamoDB, Boto3, aiobotocore, or s3fs in conjunction would start breaking with a new major version and you'd still have a urllib3 pin somewhere in that set. We'd also be detracting time from fixing this issue in the current major version of botocore by having to maintain multiple variants of this package. There's no tangible benefit to the end user in that setup.
Thanks for your continued patience while this is addressed.
This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
To follow up, this was resolved in today's release (botocore 1.34.63) with #3141. Users on Python 3.10+ should be free to control their urllib3 2.x requirements going forward.