Use of Generic and TypeVar

Question

I'm not able to understand the use of Generic and TypeVar, and how they are related. https://docs.python.org/3/library/typing.html#building-generic-types

The docs have this example:

class Mapping(Generic[KT, VT]):
    def __getitem__(self, key: KT) -> VT:
        ...
        # Etc.

---

X = TypeVar('X')
Y = TypeVar('Y')

def lookup_name(mapping: Mapping[X, Y], key: X, default: Y) -> Y:
    try:
        return mapping[key]
    except KeyError:
        return default

Type variables exist primarily for the benefit of static type checkers. They serve as the parameters for generic types as well as for generic function definitions.

What can't I simply use Mapping with some existing type, like int, instead of creating X and Y?

Answer 1

Type variables are literally "variables for types". Similar to how regular variables allow code to apply to multiple values, type variables allow code to apply to multiple types.
At the same time, just like code is not required to apply to multiple values, it is not required to depend on multiple types. A literal value can be used instead of variables, and a literal type can be used instead of type variables – provided these are the only values/types applicable.

Since the Python language semantically only knows values – runtime types are also values – it does not have the facilities to express type variability. Namely, it cannot define, reference or scope type variables. Thus, typing represents these two concepts via concrete things:

• A typing.TypeVar represents the definition and reference to a type variable.
• A typing.Generic represents the scoping of types, specifically to class scope.

Notably, it is possible to use TypeVar without Generic – functions are naturally scoped – and Generic without TypeVar – scopes may use literal types.

---

Consider a function to add two things. The most naive implementation adds two literal things:

def add():
    return 5 + 12

That is valid but needlessly restricted. One would like to parameterise the two things to add – this is what regular variables are used for:

def add(a, b):
    return a + b

Now consider a function to add two typed things. The most naive implementations adds two things of literal type:

def add(a: int, b: int) -> int:
    return a + b

That is valid but needlessly restricted. One would like to parameterise the types of the two things to add – this is what type variables are used for:

T = TypeVar("T")

def add(a: T, b: T) -> T:
    return a + b

Now, in the case of values we defined two variables – a and b but in the case of types we defined one variable – the single T – but used for both variables! Just like the expression a + a would mean both operands are the same value, the annotation a: T, b: T means both parameters are the same type. This is because our function has a strong relation between the types but not the values.

---

While type variables are automatically scoped in functions – to the function scope – this is not the case for classes: a type variable might be scoped across all methods/attributes of a class or specific to some method/attribute.

When we define a class, we may scope type variables to the class scope by adding them as parameters to the class. Notably, parameters are always variables – this applies to regular parameters just as for type parameters. It just does not make sense to parameterise a literal.

#       v value parameters of the function are "value variables"
def mapping(keys, values):
    ...

#       v type parameters of the class are "type variables"
class Mapping(Generic[KT, VT]):
    ...

When we use a class, the scope of its parameters has already been defined. Notably, the arguments passed in may be literal or variable – this again applies to regular arguments just as for type arguments.

#       v pass in arguments via literals
mapping([0, 1, 2, 3], ['zero', 'one', 'two', 'three'])
#       v pass in arguments via variables
mapping(ks, vs)

#          v pass in arguments via literals
m: Mapping[int, str]
#          v pass in arguments via variables
m: Mapping[KT, VT]

Whether to use literals or variables and whether to scope them or not depends on the use-case. But we are free to do either as required.

Answer 2

The whole purporse of using Generic and TypeVar (here represented as the X and Y variables) is when one wants the parameters to be as generic as possible. int can be used, instead, in this case. The difference is: the static analyzer will interpret the parameter as always being an int.

Using generics mean the function accepts any type of parameter. The static analyzer, as in an IDE for instance, will determine the type of the variables and the return type as the arguments are provided on function call or object instantiation.

---

mapping: Mapping[str, int] = {"2": 2, "3": 3}
name = lookup_name(mapping, "1", 1)

In the above example type checkers will know name will always be an int relying on the type annotations. In IDEs, code completion for int methods will be shown as the 'name' variable is used.

Using specific types is ideal if that is your goal. The function accepting only a map with int keys or values, and/or returning int in this case, for instance.

As X and Y are variable you can choose any name want, basically.

Below example is possible:

def lookup_name(mapping: Mapping[str, int], key: str, default: int) -> int:
    try:
        return mapping[key]
    except KeyError:
        return default

The types are not generic in the above example. The key will always be str; the default variable, the value, and the return type will always be an int. It's the programmer's choice. This is not enforced by Python, though. A static type checker like mypy is needed for that.

The Generic type could even be constrained if wanted:

import typing

X = typing.TypeVar("X", int, str) # Accept int and str
Y = typing.TypeVar("Y", int) # Accept only int

@MisterMiyagi's answer offers a thorough explanation on the use scope for TypeVar and Generic.