Classmethods in Generic Protocols with self-types, mypy type checking failure

Question

A little background, I essentially need to define an int wrapper type, say MyInt (among some other classes), and another generic Interval type which can accept MyInt objects as well as other types of objects. Since the types acceptable by the Interval do not fall into a neat hierarchy, I thought this would be a perfect use-case for the experimental Protocol, which in my case would require a couple of methods and a couple of @classmethods. All the methods return a "self-type", i.e., MyInt.my_method returns a MyInt. Here is a MCVE:

from dataclasses import dataclass
from typing import Union, ClassVar, TypeVar, Generic, Type

from typing_extensions import Protocol


_P = TypeVar('_P', bound='PType')
class PType(Protocol):
    @classmethod
    def maximum_type_value(cls: Type[_P]) -> _P:
        ...
    @classmethod
    def minimum_type_value(cls: Type[_P]) -> _P:
        ...
    def predecessor(self: _P) -> _P:
        ...
    def successor(self: _P) -> _P:
        ...

@dataclass
class MyInteger:
    value: int
    _MAX: ClassVar[int] = 42
    _MIN: ClassVar[int] = -42
    def __post_init__(self) -> None:
        if not (self._MIN <= self.value <= self._MAX):
            msg = f"Integers must be in range [{self._MIN}, {self._MAX}]"
            raise ValueError(msg)
    @classmethod
    def maximum_type_value(cls) -> MyInteger:
        return MyInteger(cls._MAX)
    @classmethod
    def minimum_type_value(cls) -> MyInteger:
        return MyInteger(cls._MIN)
    def predecessor(self) -> MyInteger:
        return MyInteger(self.value - 1)
    def successor(self) -> MyInteger:
        return MyInteger(self.value + 1)


@dataclass
class Interval(Generic[_P]):
    low: _P
    high: _P

interval = Interval(MyInteger(1), MyInteger(2))
def foo(x: PType) -> PType:
    return x
foo(MyInteger(42))

However, mypy complains:

(py37) Juans-MacBook-Pro: juan$ mypy mcve.py
mcve.py:46: error: Value of type variable "_P" of "Interval" cannot be "MyInteger"
mcve.py:49: error: Argument 1 to "foo" has incompatible type "MyInteger"; expected "PType"
mcve.py:49: note: Following member(s) of "MyInteger" have conflicts:
mcve.py:49: note:     Expected:
mcve.py:49: note:         def maximum_type_value(cls) -> <nothing>
mcve.py:49: note:     Got:
mcve.py:49: note:         def maximum_type_value(cls) -> MyInteger
mcve.py:49: note:     Expected:
mcve.py:49: note:         def minimum_type_value(cls) -> <nothing>
mcve.py:49: note:     Got:
mcve.py:49: note:         def minimum_type_value(cls) -> MyInteger

Which to me is hard to understand. Why is return-type expecting <nothing>? I tried simply not annotating cls in the protocol:

_P = TypeVar('_P', bound='PType')
class PType(Protocol):
    @classmethod
    def maximum_type_value(cls) -> _P:
        ...
    @classmethod
    def minimum_type_value(cls) -> _P:
        ...
    def predecessor(self: _P) -> _P:
        ...
    def successor(self: _P) -> _P:
        ...

However, mypy complains with a similar error message:

mcve.py:46: error: Value of type variable "_P" of "Interval" cannot be "MyInteger"
mcve.py:49: error: Argument 1 to "foo" has incompatible type "MyInteger"; expected "PType"
mcve.py:49: note: Following member(s) of "MyInteger" have conflicts:
mcve.py:49: note:     Expected:
mcve.py:49: note:         def [_P <: PType] maximum_type_value(cls) -> _P
mcve.py:49: note:     Got:
mcve.py:49: note:         def maximum_type_value(cls) -> MyInteger
mcve.py:49: note:     Expected:
mcve.py:49: note:         def [_P <: PType] minimum_type_value(cls) -> _P
mcve.py:49: note:     Got:
mcve.py:49: note:         def minimum_type_value(cls) -> MyInteger

Which to me, makes even less sense. Note, if I make these instance methods:

_P = TypeVar('_P', bound='PType')
class PType(Protocol):
    def maximum_type_value(self: _P) -> _P:
        ...
    def minimum_type_value(self: _P) -> _P:
        ...
    def predecessor(self: _P) -> _P:
        ...
    def successor(self: _P) -> _P:
        ...

@dataclass
class MyInteger:
    value: int
    _MAX: ClassVar[int] = 42
    _MIN: ClassVar[int] = -42
    def __post_init__(self) -> None:
        if not (self._MIN <= self.value <= self._MAX):
            msg = f"Integers must be in range [{self._MIN}, {self._MAX}]"
            raise ValueError(msg)
    def maximum_type_value(self) -> MyInteger:
        return MyInteger(self._MAX)
    def minimum_type_value(self) -> MyInteger:
        return MyInteger(self._MIN)
    def predecessor(self) -> MyInteger:
        return MyInteger(self.value - 1)
    def successor(self) -> MyInteger:
        return MyInteger(self.value + 1)

Then, mypy doesn't complain at all:

I've read about self-types in protocols in PEP 544, where it gives the following example:

C = TypeVar('C', bound='Copyable')
class Copyable(Protocol):
    def copy(self: C) -> C:

class One:
    def copy(self) -> 'One':
        ...

T = TypeVar('T', bound='Other')
class Other:
    def copy(self: T) -> T:
        ...

c: Copyable
c = One()  # OK
c = Other()  # Also OK

Furthermore, in PEP484, regarding typing classmethods, we see this example:

T = TypeVar('T', bound='C')
class C:
    @classmethod
    def factory(cls: Type[T]) -> T:
        # make a new instance of cls

class D(C): ...
d = D.factory()  # type here should be D

What is wrong with my Protocol / class definition? Am I missing something obvious? I would appreciate any specific answers about why this is failing, or any work-around. But note, I need these attributes to be accessible on the class.

Note, I've tried using a ClassVar, but that introduced other issues... namely, ClassVar does not accept type-variables as far as I can tell ClassVar's cannot be generic. And ideally, it would be a @classmethod since I might have to rely on other meta-data I would want to shove in the class.

Answer 1

I'm not on expert on Mypy but have been teaching myself to use it recently and I think this may be due to an issue in Mypy mentioned here:

https://github.com/python/mypy/issues/3645

The issue is with handling TypeVar variables in class methods rather than anything directly to do with protocols.

The following minimal example is given in the link to show the problem.

T = TypeVar('T')

class Factory(Generic[T]):
    def produce(self) -> T:
        ...
    @classmethod
    def get(cls) -> T:
        return cls().produce()

class HelloWorldFactory(Factory[str]):
    def produce(self) -> str:
        return 'Hello World'

reveal_type(HelloWorldFactory.get())  # mypy should be able to infer 'str' here

The output from reveal_type is T rather than str. The same thing is happening with your code, where Mypy is failing to infer the type should be MyInteger rather than _P and so doesn't see your class as implementing the protocol. Changing the return type of the class methods to 'PType' makes the errors go away, but I'm not confident enough to know if there are any other impacts of that change.

There's been some discussion on how best to handle it, because it's not trivial to decide what the correct behaviour should be in every case, so might be no harm flagging this to them for more use case examples (see https://github.com/python/mypy/issues/5664 for example.)