Typing hint for abstract class method that returns class instance

Question

I'm getting type checker errors on the following code, I would love to understand how to resolve the error.

The following base class has an abstract class method, I want that every child class that inherits from it will implement a decode function that returns an instance of the child class.

from abc import ABC, abstractmethod
from typing import TypeVar


TMetricBase = TypeVar("TMetricBase", bound="MetricBase")


class MetricBase(ABC):
    @abstractmethod
    def add(self, element: str) -> None:
        pass  # pragma: no cover

    @classmethod
    @abstractmethod
    def decode(cls, json_str: str) -> TMetricBase:
        pass  # pragma: no cover

Child class looks like the following

import json
from typing import Any, Callable, List, Mapping, Optional
from something import MetricBase, TMetricBase


class DiscreteHistogramMetric(MetricBase):
    def __init__(self, histogram: Optional[Mapping[str, int]]) -> None:
        super().__init__()
        self._histogram = dict(histogram) if histogram else {}

    def add(self, element: str) -> None:
        self._histogram[element] = self._histogram.get(element, 0) + 1

    @classmethod
    def decode(cls, json_str: str) -> "DiscreteHistogramMetric":
        json_obj = json.loads(json_str)
        histogram_map = json_obj["DiscreteHistogramMetric"]
        return cls(histogram=histogram_map)

I'm getting the following error:

error: Return type of "decode" incompatible with supertype "MetricBase"

When changing decode's return type to TMetricBase, I get the following error:

error: Incompatible return value type (got "DiscreteHistogramMetric", expected "TMetricBase")

Answer 1

The error has to do with how you're having just a single TypeVar in the return type of decode. It's unclear what that would mean, exactly -- you're more or less trying to declare that every single subclass of MetricBase needs to support returning any other arbitrary subclass of MetricBase, which it'll somehow magically infer based on how that function is being called.

This isn't really something that's possible to do in Python.

What you'll need to do instead is one of the following:

1. Give up and not use TypeVars
2. Make MetricBase a generic class and have your subclasses inherit a parameterized version of MetricBase.
3. Use TMetricBase in the decode parameters in some way. (This way, we can actually deduce what the return type ought to be).

I'm assuming you've already considered the first solution and rejected it: it would make our program type check, but would also make the decode method somewhat useless/require some clunky casting.

The second solution looks something like this:

from abc import ABC, abstractmethod
from typing import TypeVar, Generic

TMetricBase = TypeVar("TMetricBase", bound="MetricBase")

class MetricBase(ABC, Generic[TMetricBase]):
    @classmethod
    @abstractmethod
    def decode(cls, json_str: str) -> TMetricBase:
        pass

class DiscreteHistogramMetric(MetricBase['DiscreteHistogramMetric']):
    @classmethod
    def decode(cls, json_str: str) -> "DiscreteHistogramMetric":
        pass

By having DiscreteHistogramMetric subclass MetricBase[DiscreteHistogramMetric] instead of just MetricBase directly, we can actually constrain the typevar to something meaningful.

This solution is still a little clunky though -- having to subclass MetricBase requires us to start using generics wherever we use MetricBase which is pretty annoying.

The third solution on the surface initially sounds even clunkier: are we going to add in some extra dummy third param or some nonsense? But it turns out there's a nice trick we can use -- we can use generic selfs to annotate the cls variable!

Normally, the type of that variable is inferred and doesn't need to be annotated, but in this case, it's helpful to do so: we can use information about what exactly cls is to help produce a more refined return type.

Here's what it looks like:

from abc import ABC, abstractmethod
from typing import TypeVar, Type

TMetricBase = TypeVar("TMetricBase", bound="MetricBase")

class MetricBase(ABC):
    @classmethod
    @abstractmethod
    def decode(cls: Type[TMetricBase], json_str: str) -> TMetricBase:
        pass

class DiscreteHistogramMetric(MetricBase):
    def __init__(self, something: str) -> None:
        pass

    @classmethod
    def decode(cls: Type[TMetricBase], json_str: str) -> TMetricBase:
        # Note that we need to use create the class by using `cls` instead of
        # using `DiscreteHistogramMetric` directly.
        return cls("blah")

It's a bit unfortunate that we need to continue using TypeVars within the subclass instead of defining it more simply the way you did in your question -- I believe this behavior is a bug in mypy.

However, it does do the trick: doing DiscreteHistogramMetric.decode("blah") will return a TMetricBase as expected.

And unlike the first approach, the messiness is at least pretty well-confined to the decode method and doesn't require you to start using generics wherever you're also using MetricBase classes.