mypy error, overload with Union/Optional, "Overloaded function signatures 1 and 2 overlap with incompatible return types"

Question

So, let's start with an example. Suppose we have several types that can be combined together, let's say we are using __add__ to implement this. Unfortunately, due to circumstances beyond our control, everything has to be "nullable", so we are forced to use Optional everywhere.

from typing import Optional, List, overload
class Foo:
    value: int
    def __init__(self, value: int) -> None:
        self.value = value
    def __add__(self, other: 'Foo') -> 'Optional[Foo]':
        result = self.value - other.value
        if result > 42:
            return None
        else:
            return Foo(result)

class Bar:
    value: str
    def __init__(self, value: str) -> None:
        self.value = value
    def __add__(self, other: 'Bar') -> 'Optional[Bar]':
        if len(self.value) + len(other.value) > 42:
            return None
        else:
            return Bar(self.value + other.value)

class Baz:
    value: List[str]
    def __init__(self, value:List[str]) -> None:
        self.value = value
    def __add__(self, other: 'Bar') -> 'Optional[Baz]':
        if len(self.value) + 1 > 42:
            return None
        else:
            return Baz([*self.value, other.value])


@overload
def Add(this: Optional[Foo], that: Optional[Foo]) -> Optional[Foo]:
    ...
@overload
def Add(this: Optional[Bar], that: Optional[Bar]) -> Optional[Bar]:
    ...
@overload
def Add(this: Optional[Baz], that: Optional[Bar]) -> Optional[Baz]:
    ...
def Add(this, that):
    if this is None or that is None:
        return None
    else:
        return this + that

We want utility function that does null-checking for us, but can generically handle "combinable" types. Most of the types can only be combined with themselves, and just to be more true to my actual use-case, let's say one type combines with the other. I would have hoped that the overload decorator could have helped here, however mypy complains:

mcve4.py:35: error: Overloaded function signatures 1 and 2 overlap with incompatible return types
mcve4.py:35: error: Overloaded function signatures 1 and 3 overlap with incompatible return types
mcve4.py:38: error: Overloaded function signatures 2 and 3 overlap with incompatible return types

Using mypy version: mypy 0.641

Note, if I remove Optional madness, mypy doesn't complain. I can even keep one of them as optional!:

from typing import List, overload
class Foo:
    value: int
    def __init__(self, value: int) -> None:
        self.value = value
    def __add__(self, other: 'Foo') -> 'Foo':
        result = self.value - other.value
        return Foo(result)

class Bar:
    value: str
    def __init__(self, value: str) -> None:
        self.value = value
    def __add__(self, other: 'Bar') -> 'Bar':
        return Bar(self.value + other.value)

class Baz:
    value: List[str]
    def __init__(self, value:List[str]) -> None:
        self.value = value
    def __add__(self, other: 'Bar') -> 'Optional[Baz]':
        return Baz([*self.value, other.value])


@overload
def Add(this: Foo, that: Foo) -> Foo:
    ...
@overload
def Add(this: Bar, that: Bar) -> Bar:
    ...
@overload
def Add(this: Baz, that: Bar) -> 'Optional[Baz]':
    ...
def Add(this, that):
    if this is None or that is None:
        return None
    else:
        return this + that

This makes me suspect that the "overlap" is for NoneType, but feel like this should be resolvable, am I completely off base?

Edit

So, I'm really just flailing about here, but I suppose, when both argument are None this is definitely ambiguous, I would have hoped that the following would resolve it:

@overload
def Add(this: None, that: None) -> None:
    ...
@overload
def Add(this: Optional[Foo], that: Optional[Foo]) -> Optional[Foo]:
    ...
@overload
def Add(this: Optional[Bar], that: Optional[Bar]) -> Optional[Bar]:
    ...
@overload
def Add(this: Optional[Baz], that: Optional[Bar]) -> Optional[Baz]:
    ...
def Add(this, that):
    if this is None or that is None:
        return None
    else:
        return this + that

But I am still getting:

mcve4.py:37: error: Overloaded function signatures 2 and 3 overlap with incompatible return types
mcve4.py:37: error: Overloaded function signatures 2 and 4 overlap with incompatible return types
mcve4.py:40: error: Overloaded function signatures 3 and 4 overlap with incompatible return types

Edit2 Going along the same garden-trail, I've tried the following:

@overload
def Add(this: None, that: None) -> None:
    ...
@overload
def Add(this: Foo, that: Optional[Foo]) -> Optional[Foo]:
    ...
@overload
def Add(this: Optional[Foo], that: Foo) -> Optional[Foo]:
    ...
@overload
def Add(this: Baz, that: Bar) -> Optional[Baz]:
    ...
@overload
def Add(this: Baz, that: Optional[Bar]) -> Optional[Baz]:
    ...
@overload
def Add(this: Optional[Baz], that: Bar) -> Optional[Baz]: # 6
    ...
@overload
def Add(this: Bar, that: Optional[Bar]) -> Optional[Bar]:
    ...
@overload
def Add(this: Optional[Bar], that: Bar) -> Optional[Bar]: # 8
    ...

def Add(this, that):
    if this is None or that is None:
        return None
    else:
        return this + that

I am now getting:

mcve4.py:49: error: Overloaded function signatures 6 and 8 overlap with incompatible return types

Which is starting to make sense to me, I think fundamentally what I am trying to do is unsafe/broken. I may have to just cut the gordian knot another way...

Answer 1

There is not going to be a particularly clean way of solving this, I'm afraid -- at least, none that I'm personally aware of. As you've observed, your type signatures contain a fundamental ambiguity that mypy will not allow: if you attempt to call Add with an argument of type None, mypy will fundamentally not be able to deduce which of the given overload variants matches.

For more discussion about this, see the mypy docs on checking overload invariants -- search for the paragraph discussing "inherently unsafely overlapping variants" and start reading from there.

---

However, we can wiggle free in this specific case by spelling out the overloads to more precisely match the actual runtime behavior. In particular, we have this nice property that if either argument is 'None', we must also return None. If we encode this, mypy ends up being satisfied:

@overload
def Add(this: None, that: None) -> None:
    ...
@overload
def Add(this: Foo, that: None) -> None:
    ...
@overload
def Add(this: Bar, that: None) -> None:
    ...
@overload
def Add(this: Baz, that: None) -> None:
    ...
@overload
def Add(this: None, that: Foo) -> None:
    ...
@overload
def Add(this: None, that: Bar) -> None:
    ...
@overload
def Add(this: Foo, that: Foo) -> Foo:
    ...
@overload
def Add(this: Bar, that: Bar) -> Bar:
    ...
@overload
def Add(this: Baz, that: Bar) -> Baz:
    ...
def Add(this, that):
    if this is None or that is None:
        return None
    else:
        return this + that

x: Optional[Baz]
y: Optional[Bar]

reveal_type(Add(x, y))  # Revealed type is 'Union[Baz, None]'

The fact that this works may initially seem surprising -- after all, we pass in an argument of type Optional[...] and yet none of the overloads contain that type!

What mypy is doing here is informally called "union math" -- it basically observes that x and y are both unions of type Union[Baz, None] and Union[Bar, None] respectively, and tries doing the spiritual equivalent of a nested for loop to check every single possible combination of these unions. So, in this case it checks for an overload variant that matches (Baz, Bar), (Baz, None), (None, Bar), and (None, None) and gets back return values of types Baz, None, None, and None respectively.

The final return type is then the union of these values: Union[Baz, None, None, None]. This simplifies to Union[Baz, None], which is the desired return type.

---

The main downside of this solution of course, is that it's extremely verbose -- perhaps to an unbearable extent, depending on how many of these helper functions you have and how pervasive this "we could maybe return None" problem is in your codebase.

If this is the case, something you could possibly do is declare "bankruptcy" on the pervasiveness of 'None' throughout your codebase and start running mypy with "strict optional" mode disabled.

In short, if you run mypy with the --no-strict-optional flag, you're instructing mypy to assume that 'None' is a valid member of every class. This is the same as how Java assumes that 'null' is a valid member of every type. (Well, every non-primitive type, but whatever).

This weakens the type safety of your code (sometimes dramatically), but it will let you simplify your code to look like this:

class Foo:
    value: int
    def __init__(self, value: int) -> None:
        self.value = value

    # Note: with strict-optional disabled, returning 'Foo' vs
    # 'Optional[Foo]' means the same thing
    def __add__(self, other: 'Foo') -> Foo:
        result = self.value - other.value
        if result > 42:
            return None
        else:
            return Foo(result)


@overload
def Add(this: Foo, that: Foo) -> Foo:
    ...
@overload
def Add(this: Bar, that: Bar) -> Bar:
    ...
@overload
def Add(this: Baz, that: Bar) -> Baz:
    ...
def Add(this, that):
    if this is None or that is None:
        return None
    else:
        return this + that

x: Optional[Baz]
y: Optional[Bar]

reveal_type(Add(x, y))  # Revealed type is 'Baz'

Strictly speaking, the overload checks ought to report the "unsafely overlapping" error for the same reason they did back when strict-optional was enabled. However, if we did so, overloads would be completely unusable when strict-optional is disabled: so mypy deliberately weakens the checks here and ignores that particular error case.

The main disadvantage of this mode is that you're now forced to do more runtime checking. If you receive back some value of type Baz, it might actually be None -- similar to how any object reference in Java could actually be null.

This might potentially be an ok tradeoff in your case, since you're already scattering these types of runtime checks everywhere.

If you subscribe to the "null was a billion-dollar mistake" school of thought and would like to live in the strict-optional world, one technique you can use is to progressively re-enable strict-optional in select parts of your codebase by using the mypy config file.

Basically, you can configure many (though not all) mypy options on a per-module basis via the config file, which can be pretty handy if you're attempting to add types to a pre-existing codebase and find that transitioning all at once is simply intractable. Start with loose global settings, then gradually make them stricter and stricter over time.

---

If both of these options feels too extreme (e.g. you don't want to add the verbose signature from above everywhere, but also don't want to give up strict optional), the final option you could do is to just simply silence the error by adding a # type: ignore to every line mypy reports an "unsafely overlapping types" error on.

This is also defeat in a way, but possibly a more localized one. Even typeshed, the repository of type hints for the standard library, contains a few scattered # type: ignore comments here and there for certain functions that are simply inexpressible using PEP 484 types.

Whether or not this is a solution you're ok will depend on your particular circumstance. If you analyze your codebase, and think that the potential unsafeness is something you're ok with ignoring, maybe this might be the simplest way forward.