mypy: how to define a generic subclass

Question

I have a subclass of queue.Queue like so:

class SetQueue(queue.Queue):
    """Queue which will allow a given object to be put once only.

    Objects are considered identical if hash(object) are identical.
    """

    def __init__(self, maxsize=0):
        """Initialise queue with maximum number of items.

        0 for infinite queue
        """
        super().__init__(maxsize)
        self.all_items = set()

    def _put(self):
        if item not in self.all_items:
            super()._put(item)
            self.all_items.add(item)

I am trying to use mypy for static type checking. In this case, the SetQueue should take a generic object T. This is my attempt so far:

from typing import Generic, Iterable, Set, TypeVar

# Type for mypy generics
T = TypeVar('T')

class SetQueue(queue.Queue):
    """Queue which will allow a given object to be put once only.

    Objects are considered identical if hash(object) are identical.
    """

    def __init__(self, maxsize: int=0) -> None:
        """Initialise queue with maximum number of items.

        0 for infinite queue
        """
        super().__init__(maxsize)
        self.all_items = set()  # type: Set[T]

    def _put(self, item: T) -> None:
        if item not in self.all_items:
            super()._put(item)
            self.all_items.add(item)

mypy throws a warning on the class definition line saying "Missing type parameters for generic type".

I think that I need a Generic[T] somewhere but every attempt that I have made throws a syntax error. All of the examples in the docs show subclassing from Generic[T] but don't subclass from any other object.

Does anyone know how to define the generic type for SetQueue?

Answer 1

The problem here is that queue.Queue does not actually not inherit from typing.Generic, but the typeshed stubs for it says that it does. This is a bit of a necessary evil until the stdlib fully buys into typing, if ever. As a result, the actual queue.Queue does not have the typing.GenericMeta metaclass that gives generic classes their __getitem__ ability at runtime:

For example, this code type-checks ok in mypy, but fails at runtime:

from typing import Generic, Iterable, Set, TypeVar, TYPE_CHECKING
import queue

# Type for mypy generics
T = TypeVar('T')


class SetQueue(queue.Queue[T]):
    """Queue which will allow a given object to be put once only.

    Objects are considered identical if hash(object) are identical.
    """

    def __init__(self, maxsize: int=0) -> None:
        """Initialise queue with maximum number of items.

        0 for infinite queue
        """
        super().__init__(maxsize)
        self.all_items = set()  # type: Set[T]

    def _put(self, item: T) -> None:
        if item not in self.all_items:
            super()._put(item)
            self.all_items.add(item)


my_queue = queue.Queue()  # type: queue.Queue[int]
my_queue.put(1)
my_queue.put('foo')  # error

my_set_queue = SetQueue()  # type: SetQueue[int]
my_set_queue.put(1)
my_set_queue.put('foo')  # error

The error raised is TypeError: 'type' object is not subscriptable, meaning that queue.Queue[T] (i.e. queue.Queue.__getitem__) is not supported.

Here's a hack to make it work at runtime as well:

from typing import Generic, Iterable, Set, TypeVar, TYPE_CHECKING
import queue

# Type for mypy generics
T = TypeVar('T')

if TYPE_CHECKING:
    Queue = queue.Queue
else:
    class FakeGenericMeta(type):
        def __getitem__(self, item):
            return self

    class Queue(queue.Queue, metaclass=FakeGenericMeta):
        pass


class SetQueue(Queue[T]):
    """Queue which will allow a given object to be put once only.

    Objects are considered identical if hash(object) are identical.
    """

    def __init__(self, maxsize: int=0) -> None:
        """Initialise queue with maximum number of items.

        0 for infinite queue
        """
        super().__init__(maxsize)
        self.all_items = set()  # type: Set[T]

    def _put(self, item: T) -> None:
        if item not in self.all_items:
            super()._put(item)
            self.all_items.add(item)


my_queue = queue.Queue()  # type: queue.Queue[int]
my_queue.put(1)
my_queue.put('foo')  # error

my_set_queue = SetQueue()  # type: SetQueue[int]
my_set_queue.put(1)
my_set_queue.put('foo')  # error

There may be a better way to patch in the metaclass. I'm curious to know if anyone comes up with a more elegant solution.

Edit: I should note that multiple inheritance did not work because class SetQueue(queue.Queue, Generic[T]) fails to relate SetQueue's T to queue.Queue's

Answer 2

Composition vs inheritance ("has a" vs "is a") can be extremely useful here since you can specify exactly what you want typing to be rather than relying on the state of typing in your intended parent classes (which may not be great).

Below is a complete implementation of SetQueue (from the question) that 100% passes mypy --strict today without any issues (or hackery). I stripped the docstrings for brevity.

from typing import Generic, TypeVar, Set, Optional
import queue

T = TypeVar('T')  # Generic for the item type in SetQueue

class SetQueue(Generic[T]):
    def __init__(self, maxsize: int=0) -> None:
        self._queue: queue.Queue[T] = queue.Queue(maxsize)
        self.all_items: Set[T] = set()

    def _put(self, item: T) -> None:
        if item not in self.all_items:
            self._queue.put(item)
            self.all_items.add(item)

    # 100% "inherited" methods (odd formatting is to condense passthrough boilerplate)
    def task_done(self)           -> None: return self._queue.task_done()
    def join(self)                -> None: return self._queue.join()
    def qsize(self)               -> int:  return self._queue.qsize()
    def empty(self)               -> bool: return self._queue.empty()
    def full(self)                -> bool: return self._queue.full()
    def put_nowait(self, item: T) -> None: return self.put(item)
    def get_nowait(self)          -> T:    return self.get()
    def get(self, block: bool = True, timeout: Optional[float] = None) -> T:
        return self._queue.get(block, timeout)
    def put(self, item: T, block: bool = True, timeout: Optional[float] = None) -> None:
        return self._queue.put(item, block, timeout)

Although composition is definitely more verbose than inheritance (since it requires defining all the passthrough methods), code clarity can be better. In addition, you don't always want all the parent methods and composition lets you omit them.

Composing like this can be particularly important today since the current state of typing in the python ecosystem (including the python standard library) isn't 100% awesome. There are essentially two parallel worlds: 1) Actual code, and 2) typing. Although you might be subclassing a great class from a code perspective, that does not necessarily equate to inheriting great (or even functional) type definitions. Composition can circumvent this frustration.