Typing spec should be clearer that type checkers are not expected to support PEP 3141

[Alex Waygood]

With regards to the numeric tower, PEP 484 states:

PEP 3141 defines Python’s numeric tower, and the stdlib module numbers implements the corresponding ABCs (Number, Complex, Real, Rational and Integral). There are some issues with these ABCs, but the built-in concrete numeric classes complex, float and int are ubiquitous (especially the latter two :-).

Rather than requiring that users write import numbers and then use numbers.Float etc., this PEP proposes a straightforward shortcut that is almost as effective: when an argument is annotated as having type float, an argument of type int is acceptable; similar, for an argument annotated as having type complex, arguments of type float or int are acceptable. This does not handle classes implementing the corresponding ABCs or the fractions.Fraction class, but we believe those use cases are exceedingly rare.

This is a very useful passage to link to, because:

• It clearly (albeit tersely) states that the PEP-3141 ABCs are problematic
• It clearly states the separate solution that PEP-484 proposes for dealing with numeric types
• It frames the PEP-484 solution in opposition and contrast to the PEP-3141 solution

The parallel passage in the typing spec currently states this:

Python’s numeric types complex, float and int are not subtypes of each other, but to support common use cases, the type system contains a straightforward shortcut: when an argument is annotated as having type float, an argument of type int is acceptable; similar, for an argument annotated as having type complex, arguments of type float or int are acceptable.

Since PEP-484 is a historical document rather than a piece of living documentation, it would be great if the typing spec could state as clearly as PEP-484 that the PEP-3141 ABCs are not the recommended way of annotating numeric types in Python, and that type checkers may not necessarily (and in fact probably won't ever) support them.

[Sergei Lebedev]

It could be also worth calling out that even though consistent subtyping/compatibility does allow int where float is expected, it is in fact unsound:

def f(x: float) -> bool:
  return x.is_integer()

f(42)  # type checks, but fails at runtime

[Alex Waygood]

It could be also worth calling out that even though consistent subtyping/compatibility does allow int where float is expected, it is in fact unsound:

def f(x: float) -> bool:
  return x.is_integer()

f(42)  # type checks, but fails at runtime

I think that's already touched on where the spec says:

Python’s numeric types complex, float and int are not subtypes of each other

But perhaps it could be more explicitly stated, yes

[Sergei Lebedev]

They are not subtypes, but they are part of the consistency (or rather the consistent subtyping) relation, which is what is normally used by type checkers for checking function calls. Am I interpreting the spec wording in the wrong way?

[Alex Waygood]

I believe the int/float/complex special-casing that type checkers perform is unique in that the special-casing only applies to parameter annotations; they are not seen by type checkers as subtypes or consistent subtypes in contexts outside of parameter annotations. I understood the spec to be referring to this unique aspect of the way these types are understood by type checkers when it said that they "are not subtypes of each other". But you're right it could be clearer; I'm not confident that my interpretation is correct.

[Sergei Lebedev]

Note that regardless of the terminology, this feature is unsound since it is not type safe generally to pass an int where a float is expected (as my admittedly artificial) snippet above demonstrates.

[Alex Waygood]

A discussion on the merits and demerits of the current behaviour of type checkers in this regard is outside of the scope of this issue.

[Eric Traut]

The typing spec is not meant to be a tutorial or reference for users of the type system. It's a specification that clarifies how type checkers should behave. As such, I think it's unnecessary (and arguably inappropriate) for it to discuss PEP 3141, since it has no bearing on the type system.

This topic could fit within the Typing Reference section of the docs. Contributions are welcome.

even though consistent subtyping/compatibility does allow int where float is expected...

As @AlexWaygood said above, that statement is subtly incorrect. The type int is not a subtype of float in the Python type system; the spec is clear that this is the case. Instead, the spec indicates that when you use float within a type expression, a type checker should implicitly interpret that expression as though the user had specified float | int. This is an important distinction.

this feature is unsound

That's not necessarily true. It depends on the type checker implementation. If a type checker assumes that int is a subtype of float (as mypy does), then it is unsound. If a type checker assumes that any annotation float implies float | int, then it's not unsound. There is a proposal in the mypy issue tracker to change mypy to do the latter, but it hasn't been implemented.

I attempted to implement this suggestion in pyright but needed to back out part of it for compatibility reasons. See this issue for details. The problem is that there's currently no way in the type system to provide an annotation that means "only float, not float | int", so methods like int.__truediv__ cannot be annotated in a way that allows a type checker to implement soundness checks without generating a bunch of false positives.

Here's an illustrative code sample that generates a runtime error. Pyright detects this error but mypy does not currently.

Code sample in pyright playground

def func1(f: float):
    if not isinstance(f, float):
        f.hex()  # Runtime error

func1(1)

[Sergei Lebedev]

Thanks for an extended reply, Eric.

The type int is not a subtype of float in the Python type system; the spec is clear that this is the case.

I think it would be useful if the spec clarified that int it is neither a subtype, nor is consistent with float. In my mind, saying that it isn't a subtype does not immediately eliminate it from the other relation.

Instead, the spec indicates that when you use float within a type expression, a type checker should implicitly interpret that expression as though the user had specified float | int.

I agree that this interpretation is sound, but I also find it confusing to treat float (and complex) as effectively a macro/syntax sugar.

Do you think we could re-evaluate this as part of the work in the spec? For example, we could explore an alternative design where literals have flexible types, so 42 would be {int, float, complex} instead of just int, and 42.0 -- {float, complex}.

[Sergei Lebedev]

Looking at

def func1(f: float):
    if not isinstance(f, float):
        f.hex()  # Runtime error

func1(1)

as a user, my first reaction would be to conclude that the body of the if is unreachable. This is true for any class, so it is natural to extend that to float.

[Eric Traut]

Do you think we could re-evaluate this as part of the work in the spec?

The spec is a living document, and the community welcomes new proposals. If this is a topic that's important to you and you'd like to champion a modification, feel free to create a new thread in the typing forum. This is an area where backward compatibility is really important, so any such proposals will be reviewed with that in mind.

[Alex Waygood]

I think it's unnecessary (and arguably inappropriate) for it to discuss PEP 3141, since it has no bearing on the type system.

I'm not sure I agree. PEP-3141 is a PEP that has been accepted for 17 years, and has not been deprecated. At runtime, Python recognises complex as a "virtual subclass" of numbers.Complex, float as a virtual subclass of numbers.Real and int as a virtual subclass of numbers.Integral. For any other stdlib module like this, we would either try to model this subtyping relationship in typeshed, or type checkers would implement special-casing to model it. It's reasonable for users to expect type checkers to understand the subtyping relationship that holds true at runtime, and it's a persistent source of surprise that type checkers do not.

I'm fully aware of the reasons why neither typeshed nor type checkers support PEP-3141, I don't believe that should change, and I don't want to restart a conversation about whether they should or not. But the decision by PEP-484 to state that PEP-3141 is a "runtime-only" numeric tower, and to instead implement a parallel "static-only" numeric tower that is understood by type checkers but is not understood by issubclass() checks at runtime -- this is, in my view, a fairly crucial design choice of the type system.

[Guido van Rossum]

I don't really have anything to add to this conversation, except that I believe that historically when we wrote and accepted PEP 484, I personally believed that we were specifying that int was a subtype of float (and float a subtype of complex). Until today I wasn't aware that the spec actually says that e.g. the type float should be interpreted, in certain contexts, as float | int. This is neither here nor there, it's just an admission of my flawed understanding of the subtleties here at the time.

[Jelle Zijlstra]

I was probably the person who cut out the mention of PEP 3141 from the spec. I agree with @erictraut above that a discussion of this topic is a better fit for the user-facing type system reference (which unfortunately we haven't yet done much work on). If all we are saying is that there are no special cases for type checkers, that doesn't feel like it's worth saying in the spec.

Eric's suggestion above about the interpretation of the float/int special case is interesting and it does seem to be a more sensible interpretation than treating int directly as a subtype of float.

[Alex Waygood]

If all we are saying is that there are no special cases for type checkers

From my perspective, there is a special case when it comes to PEP-3141. For all other stdlib ABCs, we've taken pains in typeshed to ensure that type checkers understand runtime virtual subclasses of those ABCs as static subtypes of those ABCs. list inherits from collections.abc.MutableSequence in typeshed, even though it doesn't at runtime; set inherits from collections.abc.MutableSet in typeshed, even though it doesn't at runtime; we pretend os.PathLike and contextlib.AbstractContextManager are protocols in typeshed, even though at runtime they're just ABCs. It's only the numbers ABCs where we deliberately don't try to model the subtyping relationship in typeshed. (And, again, I support that policy.)

[Sergei Lebedev]

I thought about this a bit more, and unfortunately flexible types have the same drawbacks as existing special-casing -- they allow for unsoundness (unless supplemented with runtime type checking).

I wonder if the "right" thing to do here would be to abandon the idea of special-casing of int, float and complex altogether? The runtime type of 42 is always an int and never float nor complex. Why allow passing 42 to an API which requires a float? Moreover, the fix is always obvious -- just add .0 or j to your literal.

[Guido van Rossum]

unsoundness (unless supplemented with runtime type checking)

But we always have runtime type checking, right? AttributeError, TypeError, and so on are exactly that. So what's the unsoundness complaint about?

[Guido van Rossum]

Why allow passing 42 to an API which requires a float?

Because Python users have been doing that for the last 30 years.

[Sergei Lebedev]

But we always have runtime type checking, right? [...]

Raising AttributeError is not the same as saying "you called function f with an int instead of a float and that caused AttributeError such and such".

Because Python users have been doing that for the last 30 years.

I agree that the type system should be designed around existing Python code, but given that it is probably impossible to account for every Python dynamic feature in a reasonable way, there will always be compromises.

Could you give an example which stops working if 42 is statically modelled as just int with no subtyping/consistent subtyping connection to float?

[Guido van Rossum]

It could be also worth calling out that even though consistent subtyping/compatibility does allow int where float is expected, it is in fact unsound:

def f(x: float) -> bool:
  return x.is_integer()

f(42)  # type checks, but fails at runtime

This passes for me, BTW. (Apparently this was a CPython bug and it was fixed in 3.12.)

The same applies to Eric's example using f.hex() -- when that was added to float it should also have been added to int. Same for other examples you may find of methods that are supported by float but not by int.

Could you give an example which stops working if 42 is statically modelled as just int with no subtyping/consistent subtyping connection to float?

This sounds like a trick question. I'd say that in 3.12 the above example works at runtime, but it seems you would like it to fail the static check? Why? Because it fails at runtime in 3.11?

[Sergei Lebedev]

This sounds like a trick question.

It was not intended as one, actually.

If we can guarantee that int <: float <: complex structurally, then there is no issue with the original wording in PEP-484, because users cannot observe the difference between the types.

[Guido van Rossum]

If we can guarantee that int <: float <: complex structurally, then there is no issue with the original wording in PEP-484, because users cannot observe the difference between the types.

That would be my preferred approach (and what I had in mind when we created PEP 484, nearly 10 years ago).

[Carl Meyer]

Perhaps it is orthogonal to the core concern of this issue, but if we amend this paragraph of the spec, I think it would also be better to use clearer terms for the relation that this special case applies to (e.g. "consistent with"), rather than discussing only argument annotations, as the current text does, which leaves underspecified whether the described behavior is also supposed to apply to non-argument annotations (e.g. x: int = 1; y: float = x).