Is PEP 585 unusable at runtime under Python 3.7 and 3.8?

Question

PEP 585 -- Type Hinting Generics In Standard Collections claims usability under both Python 3.7 and 3.8 with a standard from __future__ import annotations preamble. Notably:

For use cases restricted to type annotations, Python files with the annotations future-import (available since Python 3.7) can parameterize standard collections, including builtins.

Starting with Python 3.7, when from __future__ import annotations is used, function and variable annotations can parameterize standard collections directly. Example:

from __future__ import annotations

def find(haystack: dict[str, list[int]]) -> int:
    ...

While the above toy example does technically parse, that's about all it does. Attempting to actually use a parametrized builtin collection at runtime under either Python 3.7 or 3.8 invariably raises the dreaded TypeError: 'type' object is not subscriptable exception:

>>> def find(haystack: dict[str, list[int]]) -> int: pass
>>> print(find.__annotations__)
{'haystack': 'dict[str, list[int]]', 'return': 'int'}
>>> eval(find.__annotations__['haystack'])
TypeError: 'type' object is not subscriptable

Note the eval() statement is the standard idiom for resolving PEP 563-style postponed annotations at runtime. ^{Don't even get me started on PEP 563.}

who you gonna believe: me or your lying PEP?

This discourages the devout Pythonista in me. PEP 585 repeatedly claims that it preserves runtime usability:

Preserving the generic type at runtime enables introspection of the type which can be used for API generation or runtime type checking. Such usage is already present in the wild.

Just like with the typing module today, the parameterized generic types listed in the previous section all preserve their type parameters at runtime:

>>> list[str]
list[str]
>>> tuple[int, ...]
tuple[int, ...]
>>> ChainMap[str, list[str]]
collections.ChainMap[str, list[str]]

Of course, none of the above works under Python 3.7 or 3.8 – regardless of whether from __future__ import annotations is enabled or not:

>>> list[str]
TypeError: 'type' object is not subscriptable
>>> tuple[int, ...]
TypeError: 'type' object is not subscriptable
>>> ChainMap[str, list[str]]
TypeError: 'type' object is not subscriptable

So PEP 585 blatantly breaks the wild and all existing attempts to introspect generic types at runtime – especially from runtime type checkers. The entire "Parameters to generics are available at runtime" section is a charade.

Am I missing something painfully obvious or are parametrized builtin collections the poison pill they superficially appear to be? Since evaluating these collections at runtime under Python 3.7 and 3.8 unconditionally raises exceptions, they're unusable at runtime – rendering them not simply useless but directly harmful for the widespread use case of type introspection and especially runtime type checking.

between a rock and a hard PEP

Any codebase type-hinting with parametrized builtin collections will be fundamentally incompatible with runtime type checkers under Python 3.7 and 3.8. Codebases preferring runtime to static type checking while preserving backward compatibility with Python < 3.9 (which has yet to even be officially released as of this writing) thus have no choice but to avoid parametrized builtin collections entirely.

Except that too is infeasible. Why? Because PEP 585 deprecates the entire hierarchy of typing pseudo-containers:

Importing those [e.g., typing.Tuple, typing.List, typing.Dict] from typing is deprecated. Due to PEP 563 and the intention to minimize the runtime impact of typing, this deprecation will not generate DeprecationWarnings. Instead, type checkers may warn about such deprecated usage when the target version of the checked program is signalled to be Python 3.9 or newer. It's recommended to allow for those warnings to be silenced on a project-wide basis.

The deprecated functionality will be removed from the typing module in the first Python version released 5 years after the release of Python 3.9.0.

Consider typing.Tuple[int], for example. By 2025 (or shortly thereafter), typing.Tuple and thus typing.Tuple[int] goes away. But tuple isn't safely parametrizable under Python 3.7 and 3.8, because doing so renders your project incompatible with anything that introspects types. So tuple[int] isn't a viable option, either.

So there are no forward- and backward-compatible options. Instead, either:

• Prohibit type introspection (and thus runtime type checking) entirely by just preferring builtin containers (e.g., tuple[int]) to typing pseudo-containers (e.g., typing.Tuple[int]) or...
• Support type introspection (and thus runtime type checking) by either:
  • Preferring typing pseudo-containers to builtin containers until 2025. At that time, both the project in question and all downstream projects of that project will need to be refactored as follows:
    • Drop Python 3.7 and 3.8 support.
    • Replace all typing pseudo-containers with builtin containers.
  • Immediately dropping Python 3.7 and 3.8 support by preferring builtin containers to typing pseudo-containers. This has the distasteful disadvantage of requiring a currently unstable Python interpreter, but... that's technically an option. Somehow.

In 2020, there are no good options – only a spectrum of increasingly horrifying lessers of several malignant evils. One would hope that PEP authors would actually test their implementations at runtime. Yet, here we are, adrift without a paddle in a steaming cesspit of theorycrafted anti-APIs. Welcome to Python.

but that's not all

There is technically a third way. It's even more distasteful – but it should technically work. One awful theorycrafting deserves another, I always say!

Since PEP 563-driven postponed annotations are merely strings, type introspection could cleverly run a regex-based replacement on each type being introspected. For each type that is a postponed annotation, globally replace each substring referencing a parametrized builtin container (e.g., list[str]) in that annotation string with the corresponding substring referencing a parametrized typing pseudo-container (e.g., List[str]).

The result? A Python 3.7- and 3.8-compatible postponed annotation string safely evaluatable until 2025, at which point that internal replacement (and Python 3.7 and 3.8 support) could just be quietly dropped.

That's a totally cray-cray ludicrous speed kludge for the stars, but... that would probably work. The core issue, of course, is that one shouldn't need insane hackery just to comply with core official PEPs. But there's an even deeper underlying cultural issue underneath that technical issue. No one – neither the author of PEP 585 nor any of the commentators reviewing PEP 585 – actually tested their new hypothetical proposed functionality before deprecating the existing well-tested functionality that actually worked.

Core official PEPs should just work out of the box. Increasingly, they don't. And that should concern everyone.

Answer 1

I'm not sure why you're posting this here on StackOverflow? If you have feedback regarding a PEP, I think you're better off posting it on either the python-dev or typing-sig mailing lists.

For example, you could perhaps try arguing that:

1. We should remove the pseudo-containers at a later date then 2025.
2. The typing_extensions module should be updated to provide shims for list and dict that replicate the runtime functionality that'll be added in Python 3.9.
3. It would be better to replace psuedo-containers in typing with aliases to builtins instead of removing them. (I suspect this will most likely be what ends up happening: e.g. see this discussion in typing-sig)

...and so forth. I'm sure you can think of more ideas, these were just ones I came up with now.

In any case, I think it should be easy enough to handle psuedo-containers being removed from typing while preserving backwards compatibility. For example, you could try:

1. Writing a tool to automatically update your code (assuming the tool hasn't already been created by somebody).

2. Monkey-patching typing to add back what's missing.

3. Switching ahead of time to using your own typing shim that does something like the following:

   from typing import *
   if sys.version_info >= (3, blah):
       List = list
       # etc
   

Granted, these approaches also aren't also perfectly clean, but they do let you achieve your goal of ensuring full backwards compatibility.

The "prohibit type introspection and switch early" approach will also likely be fairly reasonable in practice: I suspect a fairly large percentage of codebases use type hints exclusively for static type analysis.

But more broadly, if you want more seamless runtime introspection of type hints, I recommend you subscribe to either typing-sig or python-dev and give your feedback whenever typing-related PEPs are proposed.

So far, I don't think anybody's cared enough about runtime introspection of type hints to do anything beyond ensuring there continues to be baseline support for them. If you're dissatisfied with this status quo, you should try stepping up and championing whatever changes you think need to be made.

After all, Python is a volunteer-driven project. So if you want to change something about Python, the best way of doing so is to volunteer your time and energy instead of waiting for others to do so on your behalf.