Using Typing and Mypy with Descriptors

Question

I have looked at a few SO posts and github issues related to using Typing with descriptors but I have not been able to get my issue resolved.

I have wrapper classes and I want to define properties as descriptos that can get and "cast" properties of an internal data structure.

class DataDescriptor(object):
    def __init__(self, name: str, type_):
        self.name = name
        self.type_ = type_

    def __get__(self, instance, cls):
        if not instance:
            raise AttributeError("this descriptor is for instances only")
        value = getattr(instance._data, self.name)
        return self.type_(value)


class City(object):
    zip_code: str = DataDescriptor("zip_code", str)
    # mypy: Incompatible types in assignment

    population: float = DataDescriptor("population", float)
    # mypy: Incompatible types in assignment

    def __init__(self, data):
        self._data = data


class InternalData:
    # Will be consumed through city wrapper
    def __init__(self):
        self.zip_code = "12345-1234"
        self.population = "12345"
        self.population = "12345"


data = InternalData()
city = City(data)
assert city.zip_code == "12345-1234"
assert city.population == 12345.0

I thought I might be able to use TypeVar but I haven't been able to wrap my head around it.

This is what I have tried - I thought I could dynamically describe that the descriptor will take a "type", and this type is also the type __get__ will return. Am I on the right track?

from typing import TypeVar, Type
T = TypeVar("T")


class DataDescriptor(object):
    def __init__(self, name: str, type_: Type[T]):
        self.name = name
        self.type_ = type_

    def __get__(self, instance, cls) -> T:
        if not instance:
            raise AttributeError("this descriptor is for instances only")
        value = getattr(instance._data, self.name)
        return self.type_(value)
        # Too many arguments for "object"mypy(error)

Answer 1

Your solution was close. In order to get it fully working, you needed to make just three more changes:

1. Make your entire DataDescriptor class generic, not just its methods.

   When you use a TypeVar within your constructor and method signatures just by itself, what you're ending up doing is making each method independently generic. This means whatever value is bound to __init__'s T will actually end up being completely independent from whatever value of T __get__ will return!

   This is the complete opposite of what you want: you want the value of T between your different methods to be exactly the same.

   To fix, have DataDescriptor inherit from Generic[T]. (At runtime, this will mostly be the same as inheriting from object.)

2. Within City, either get rid of the type annotations for your two fields or annotate them as being of type DataDescriptor[str] and DataDescriptor[float] respectively.

   Basically, what's happening here is that your fields themselves are actually DataDescriptor objects, and need to be annotated as such. Later, when you actually try using your city.zip_code and city.population fields, mypy will realize that those fields are descriptors, and make their types be whatever the return type of your __get__ method is.

   This behavior corresponds to what happens at runtime: your attributes are actually descriptors, and you get back either a float or a str only when you try accessing those attributes.

3. Within the signature for DataDescriptor.__init__, change Type[T] to either Callable[[str], T], Callable[[Any], T], or Callable[[...], T].

   Basically, the reason why doing Type[T] doesn't work is that mypy doesn't know exactly what kind of Type[...] object you might be giving your descriptor. For example, what would happen if you tried doing foo = DataDescriptor('foo', object)? This would make __get__ end up calling object("some value"), which would crash at runtime.

   So instead, let's have your DataDescriptor accept any kind of converter function. Depending on what you want, you could have your converter function accept only a string (Callable[[str], T]), any single argument of any arbitrary type (Callable[[Any], T]), or literally any number of arguments of any arbitrary type (Callable[..., T]).

Putting these all together, your final example will look like this:

from typing import Generic, TypeVar, Any, Callable

T = TypeVar('T')

class DataDescriptor(Generic[T]):
    # Note: I renamed `type_` to `converter` because I think that better
    # reflects what this argument can now do.
    def __init__(self, name: str, converter: Callable[[str], T]) -> None:
        self.name = name
        self.converter = converter

    def __get__(self, instance: Any, cls: Any) -> T:
        if not instance:
            raise AttributeError("this descriptor is for instances only")
        value = getattr(instance._data, self.name)
        return self.converter(value)


class City(object):
    # Note that 'str' and 'float' are still valid converters -- their
    # constructors can both accept a single str argument.
    #
    # I also personally prefer omitting type hints on fields unless
    # necessary: I think it looks cleaner that way.
    zip_code = DataDescriptor("zip_code", str)
    population = DataDescriptor("population", float)

    def __init__(self, data):
        self._data = data


class InternalData:
    def __init__(self):
        self.zip_code = "12345-1234"
        self.population = "12345"
        self.population = "12345"


data = InternalData()
city = City(data)

# reveal_type is a special pseudo-function that mypy understands:
# it'll make mypy print out the type of whatever expression you give it.
reveal_type(city.zip_code)    # Revealed type is 'str'
reveal_type(city.population)  # Revealed type is 'float'