Support the `typing-extensions` backport
gsingh93 opened this issue · comments
Is your feature request related to a problem? Please describe.
I'm using backported features from https://pypi.org/project/typing-extensions/ in older version of Python, but the tool doesn't seem to support this backport.
Describe the solution you'd like
Support for the typing-extensions backport
If I have the following code:
from typing import TypedDict
vermin will tell me that the minimum version is Python 3.8. But if I do this:
from typing_extensions import TypedDict
I get this:
Minimum required versions: ~2, ~3
Note: Not enough evidence to conclude it won't work with Python 2 or 3.
I would expect this to say at least 3.6.
Thanks for reporting this. I'm not familiar with that backport but it sounds like it might be similar to the typing backport?
The first code, though, with --backport typing, the result is:
'typing' module requires 2.7, 3.2
'typing.TypedDict' member requires 2.7, 3.8
Instead of the following without:
'typing' module requires !2, 3.5
'typing.TypedDict' member requires !2, 3.8
I can see it's different. I will have a look into it.
Yea, it's similar to typing, but according to the docs for typing: "NOTE: in Python 3.5 and later, the typing module lives in the stdlib, and installing this package has NO EFFECT, because stdlib takes higher precedence than the installation directory". And according to the GH page:
Historically, this repository also hosted:
The typing_extensions package, which now lives in the [typing_extensions](https://github.com/python/typing_extensions) repo. It used to be in the typing_extensions directory.
A backport of the [typing module](https://docs.python.org/3/library/typing.html) for older Python versions. It was removed after all Python versions that lack typing in the standard library reached end of life. The last released version, supporting Python 2.7 and 3.4, is [available at PyPI](https://pypi.org/project/typing/)."
So it seems like typing won't be helpful for features added in 3.8 (for example), when used with Python 3.6, but typing_extensions should work for that.
That's interesting. Though they state on their GH that it's Python 3.7+:
typing_extensionssupports Python versions 3.7 and higher. In the future, support for older Python versions will be dropped some time after that version reaches end of life.
Not 3.6.
Ah, yea, but I think there are older versions on PyPi that will at least support typing features up to Python 3.10 for versions back to 3.6, i.e. pip install typing-extensions==4.0.0. They just won't get access to any newer features in 3.11.
I cannot support multiple versions of packages, unfortunately. But at this point it seems to be 3.7 so I can add that. I have a quick version in a branch that will now yield the following for your code via --backport typing_extensions:
'typing_extensions' module requires !2, 3.7
Do you know if everything in typing_extensions follow that 3.7 is the minimum version, or can some of them require a higher version, like 3.8 or 3.9? If it's all 3.7 then it's easy to add the backport.
Otherwise, I could add like typing_extensions_4.0.0 to be 3.6, and typing_extensions_4.3.0 to be 3.7. 🤔
I cannot support multiple versions of packages, unfortunately.
No problem, I understand. I only care about 3.6 because Ubuntu 18.04 uses it and that won't be EOL until next year, but even having the tool output 3.7 as the minimum version will be useful.
Do you know if everything in typing_extensions follow that 3.7 is the minimum version, or can some of them require a higher version, like 3.8 or 3.9?
I'm unfortunately not an expert in this area, so I can't give a definitive answer. But my understanding is that all the features in that library should be supported on 3.7+.
Otherwise, I could add like typing_extensions_4.0.0 to be 3.6, and typing_extensions_4.3.0 to be 3.7
I don't think this is necessary right now, but it might be good to think about how to do this more generically going forward, i.e. when 3.7 is EOL.
I cannot support multiple versions of packages, unfortunately.
No problem, I understand. I only care about 3.6 because Ubuntu 18.04 uses it and that won't be EOL until next year, but even having the tool output 3.7 as the minimum version will be useful.
Sure, I can understand that.
Do you know if everything in typing_extensions follow that 3.7 is the minimum version, or can some of them require a higher version, like 3.8 or 3.9?
I'm unfortunately not an expert in this area, so I can't give a definitive answer. But my understanding is that all the features in that library should be supported on 3.7+.
That's fair. I think it's all 3.7.
Otherwise, I could add like typing_extensions_4.0.0 to be 3.6, and typing_extensions_4.3.0 to be 3.7
I don't think this is necessary right now, but it might be good to think about how to do this more generically going forward, i.e. when 3.7 is EOL.
If I don't add several versions, then I cannot support the old version that you want (3.6) since it would be the newest one (3.7). :-)
So if running with your code (-vv):
~2, ~3 examples/typing_extensions.py
'typing_extensions' module requires ~2, ~3
Tips:
- You're using potentially backported modules: typing_extensions_4.3, typing_extensions_4.0
If so, try using some of the following for better results: --backport typing_extensions_4.3 --backport typing_extensions_4.0
With -vv --backport typing_extensions_4.0:
!2, 3.6 examples/typing_extensions.py
'typing_extensions' module requires !2, 3.6
With -vv --backport typing_extensions_4.3:
!2, 3.7 examples/typing_extensions.py
'typing_extensions' module requires !2, 3.7
It's still a little rough but that'd do the job, right?
If I don't add several versions, then I cannot support the old version that you want (3.6) since it would be the newest one
Yea, I guess my concern is whether you're OK with doing this going forward? Let's say 3.7 is now EOL, would we then change --backport typing_extensions to return 3.8 instead of 3.7, and add something like --backport typing_extensions=4.X.X (whatever the last support version for 3.7 is) to return 3.7? I don't know the internal details of the tool or how much work it is to support a different version of typing_extensions, but this would end up happening each time another version is EOL.
And let's say Python 3.12 comes out with some new typing features that get backported to 3.8, but not 3.7 (which is EOL in this example). How much extra logic do you need to add to the code to make sure that Python 3.11 features still report a minimum version of 3.8, but these new Python 3.12 features still report 3.12 (as typing_extensions=4.X.X wouldn't have added support for any of these new features, you would need to upgrade, which would then increase the minimum version). I don't know how much extra logic we would have to add for each release to maintain this.
But you know the tool better than I do, so let me know if that makes sense.
Those are very good points. I don't want to do too much maintenance and keep up-to-date with tons of package versions.
Might prefer to only support the latest version. Because if I start supporting versions then others will want multiple versions of other backports, too. So updating the most current backport version would be easier (either by finding out myself or others reporting issues/PRs).
Having multiple versions does require extra code and for each version I need to do custom code. But it is possible, of course.
Btw. I like your format with typing_extensions=x.y.z better than what I just used.
So in this case, we could do like this:
typing_extensions=4.0 - https://pypi.org/project/typing-extensions/4.0.0/
typing_extensions=4.3 - https://pypi.org/project/typing-extensions/4.3.0/
typing_extensions - https://pypi.org/project/typing-extensions/4.3.0/
So if using without versioning, it's just the newest one.
Have to think a little about this.
Can you share the branch you've made these changes on? I have some ideas, but it would help to see how you added support for this.
I think there are two problems that need to be solved here:
Python EOL
When a Python version (let's say 3.6) becomes EOL and support for that Python version is dropped from typing-extensions, the developer should be able to pin their code to a specific version of typing-extensions, use --backport typing-extensions=4.X.X, and have the correct minimum version of typing features be reported (in this case 3.6). Any users doing --backport typing-extensions will see their minimum version bumped to 3.7 automatically.
It shouldn't be difficult to maintain which typing-extensions packages work with which versions of Python. We would just need to map each version from 3.5 and later to one typing-requests version, and keep that updated once a year when another Python version is EOL. It also might be possible to do this automatically with pip or PyPi.
Then, if the code sees that a feature is for 3.8 for example and we're using a typing-extensions backport, it will report the minimum version it knows about for that package version. This would give 3.7 with the latest version of the package, and 3.6 with the older version of the package.
New features are added to typing-extensions
The more complex problem is when new features are added to typing-extensions. If Python 3.6 is EOL and a feature from 3.11 is added to the latest version of typing-extensions, using only the approach described above the minimum version will be reported as 3.7 with --backport typing-extensions which is correct, but 3.6 with --backport typing-extensions=4.X.X, which is incorrect, as it should be 3.11.
If we wanted to support this use-case, we could track not only what the last version of typing-extensions supporting an EOL Python is, but also what the latest version of Python it supported at that time was. So in this case, typing-extensions=4.X.X would map to (3.6, 3.10) as the minimum/maximum versions. Then if a feature is supported by any version in that range, we can safely return the minimum version of 3.6. But if the feature is marked as 3.11, it would be out of that range and 3.11 would be returned.
Hopefully that makes sense. I think the approach above seems easy to maintain, with a small update to some dictionary mapping once a year, and it also might be possible to automate. But if this won't work with how vermin is currently designed, let me know.
Yes, I understand what you mean. However, Vermin does not concern itself with maximum versions, only minimum versions. I think we're going to try the solution we talked about by having a dictionary of versioned backports and the backport without version is the most current one. And only having versioned backports when the minimum Python versions change - I don't want to have every version of every backport. Then we'll keep that up-to-date as well as can be.
I've implemented the first version in branch typing-extensions-backport. It's still WIP.
@gsingh93 do you mind testing the feature branch? Otherwise I'm going to merge it soon. Thanks.
I just tested it, it works well. Thanks for implementing this! A few notes:
- Can we change the format to use a double equals instead of single equals? This is the convention that
pipuses (i.e.pip install typing-extensions==4.0.0) so I think it would be nice to be consistent. - We need some docs that this exists and how to use it with the different versions of the package
- This will be some extra work and isn't super important, but I think it would be a nice quality of life improvement: Currently we can only do
typing_extensions=4.0andtyping_extensions=4.3, I think it will be more convenient and potentially less confusing for people if we also allow values in-between versions (i.e.4.1) and just round up to the next defined version (4.3) because that's essentially what the user wants, and they won't have to go understand the difference between these versions. And if the version is greater than the last version we have in the dictionary (i.e.4.4), then we can round down to4.3.
Thanks! Yeah, I will change it to use == instead.
Regarding docs, the README, --help, the sample config, and the contribution docs already mention backports and that --help shows the full list. Also, when using a module that is a known backport, it'll show in the tips output section. That isn't enough?
The reasoning is to only add backports where the minimum Python versions change since it would be too much work otherwise. Right now, the version part is simply any string value because not all projects use the same versioning scheme, like semantic versioning. I think instead of trying to interpret the versions, I'll show an error if the specified backport isn't known, but list all the known ones of the backport. Like if using arguments --backport typing_extensions==4.1 it might show a message asking if they meant one of the following:
typing_extensions==4.0 - https://pypi.org/project/typing-extensions/4.0.0/ (!2, 3.6)
typing_extensions==4.3 - https://pypi.org/project/typing-extensions/4.3.0/ (!2, 3.7)
typing_extensions - https://pypi.org/project/typing-extensions/4.3.0/ (!2, 3.7)
Ah, I didn't realize that even the == versions of the backports would show up in --help, in that case it's not necessary to add any more documentation.
Your idea about raising an error if the version is incorrect sounds good to me.
Alright, now we also have this:
% ./vermin.py --backport unknown
Unknown backport: unknown
Get the full list of backports via `--help`.
% ./vermin.py --backport typing_extensions==4.1
Unknown backport: typing_extensions==4.1
Did you mean one of the following?
typing_extensions - https://pypi.org/project/typing-extensions/4.3.0/ (!2, 3.7)
typing_extensions==4.3 - https://pypi.org/project/typing-extensions/4.3.0/ (!2, 3.7)
typing_extensions==4.0 - https://pypi.org/project/typing-extensions/4.0.0/ (!2, 3.6)
Get the full list of backports via `--help`.
Thanks for the ideas and valuable feedback!