Python Typing with Exception Handling

Question

The following code is stored in a file called sample.py.

import re
from typing import Optional, Tuple
 
def func(path: str) -> Optional[Tuple[str, str]]:
    regex = re.compile(r"/'([^/']+?)'/'([^/']+?)'")
    try:
        return regex.match(path).groups()
    except AttributeError:
        return None

The Mypy Python linter throws the following error when analyzing the code:

sample.py:8: error: Incompatible return value type (got "Union[Sequence[str], Any]", expected "Optional[Tuple[str, str]]")
sample.py:8: error: Item "None" of "Optional[Match[str]]" has no attribute "groups"

While regex.match(path).groups() may return a None type, which does not have a groups attribute, the resulting exception is handled and the handling is specified in the return type. However, Mypy does not seem to understand that the exception is being handled. As far as I understand Optional[Tuple[str, str]] is the correct return type and Mypy instead insists that the less specific type Union[Sequence[str], Any] is correct . What is the proper way to use exception handling with Python typing? (Please, note that I am not asking for alternate ways to write the code without using exception handling. I am just trying to provide a minimal and complete example, where Python type checkers do not behave as  I would expect with exception handling.)

Answer 1

Mypy does not really understand exceptions on a deep level -- in this case, does not understand that since you're catching the AttributeError, it can ignore the "what if regex.match(path) is None?" case.

More generally, the fundamental assumption mypy makes is that when you have some object foo with type Union[A, B] and you do foo.bar(), both types A and B have a bar() method.

If only one of those types have a bar() method, you'll need to do one of several things:

1. Give mypy sufficient information to narrow down the union to just one of the relevant types before performing the attribute access. For example, isinstance checks, x is not None checks...
2. Acknowledge that you are attempting to do something the type checker does not understand and settle for suppressing the generated error. For example, you could cast the type, add on a # type: ignore comment, find a way of making foo be the dynamic Any type...
3. Find a way of redesigning your code to side-step this issue altogether.

(In this particular case, I suppose another alternative might be to submit a pull request to mypy adding support for this pattern. But I'm not sure if this is really feasible: changing any kind of fundamental assumption is difficult work on multiple dimensions.)

Similarly, Mypy also does not understand regexes on a deep level -- e.g. doesn't try and analyze your regex to determine how many groups you'll get and so won't understand your particular regex happens to match strings with exactly two groups. The best it can do is assert that the group will return some unknown number of strings -- hence the type Sequence[str] instead of Tuple[str, str].

This sort of limitation is pretty common in type checkers in general, actually: most type systems in mainstream languages don't really support a way to predicate a return type based on the contents of any actual values passed in. Such type systems (dependent type systems, refinement type systems...) are pretty difficult to implement and often have a steep learning curve for end users.

However, it would be easier to make mypy support this on a best-effort basis by writing a mypy plugin, if you're up for it. Specifically, try taking a look at get_method_hook() and get_function_hook().