Support `python_requires`
asottile opened this issue · comments
Similar to the cryptographic hashes, this is something else that pypi now exposes on the <a ...> tags
For instance from public pypi:
$ curl --silent https://pypi.org/simple/pyupgrade/ | grep 'data-requires-python'
<a href="https://files.pythonhosted.org/packages/c3/e7/a47529e037e71e85e9dc2726aa3fe3946640b1d51fe5136cfcaac4d64647/pyupgrade-1.12.0-py2.py3-none-any.whl#sha256=a91934fe4af06cc0b41c1a004a247992e4287d49062fd0f6c3901e1b108c984e" data-requires-python=">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*">pyupgrade-1.12.0-py2.py3-none-any.whl</a><br/>
<a href="https://files.pythonhosted.org/packages/24/04/955140415f72ab913371afceee578b691cb16217d6cdbc656249cdcc8d1d/pyupgrade-1.12.0.tar.gz#sha256=f526aea35607e20ea317c5bf88a098f0539c11330a30ea47d134810b3ef72c8b" data-requires-python=">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*">pyupgrade-1.12.0.tar.gz</a><br/>
This bit us today, so I'd like this to happen.
Just adding some context: this is kind of a difficult thing to do because it (in theory) requires reading the Python metadata from the uploaded packages, while one of the core ideas behind dumb-pypi is that you don't need to have access to the packages during generation time.
The easiest approach is probably to delegate the hard part (getting the metadata) to the person calling dumb-pypi, and then accept it as another entry in the JSON package input interface (see #21 for details). This would be equivalent to how we're handling hashes, for example.
That's an easy change in dumb-pypi but the change to the caller (which is not part of this project) will be a little harder, depending on how the caller is getting the list of packages. If the list came from a database it would be fairly easy (add another column to the database) but when it's coming from listing an S3 bucket, there isn't a great place to fetch that metadata from. The "obvious" option for S3 is to attach this as metadata on the object in S3, but the problem is that the S3 API doesn't provide metadata in response to ListObjects, but only if you make another HeadObject call for every single object individually, which will be orders of magnitude slower. This is the same reason it's hard to make hashes work when using S3 as the backing store.
we hacked around this at lyft by encoding that data into empty s3 keys (such that ListObjects still works)
{pkgname}%{sha256}%... for example
Released in v1.5.0