How to create a type that is closed under inherited operations?

Question

In the mathematical sense, a set (or type) is closed under an operation if the operation always returns a member of the set itself.

This question is about making a class that is closed under all operations inherited from its superclasses.

Consider the following class.

class MyInt(int):
    pass

Since __add__ has not been overridden, it is not closed under addition.

x = MyInt(6)
print(type(x + x))  # <class 'int'>

One very tedious way to make the type closed would be to manually cast back the result of every operation that returns an int to MyInt.

Here, I automated that process using a metaclass, but this seems like an overly complex solution.

import functools

class ClosedMeta(type):
    _register = {}

    def __new__(cls, name, bases, namespace):
        # A unique id for the class
        uid = max(cls._register) + 1 if cls._register else 0

        def tail_cast(f):
            @functools.wraps(f)
            def wrapper(*args, **kwargs):
                out = f(*args, **kwargs)
                if type(out) in bases:
                    # Since the class does not exist yet, we will recover it later
                    return cls._register[uid](out)
                return out
            return wrapper

        for base in reversed(bases):
            for name, attr in base.__dict__.items():
                if callable(attr) and name not in namespace:
                    namespace[name] = tail_cast(attr)

        subcls = super().__new__(cls, name, bases, namespace)
        cls._register[uid] = subcls
        return subcls

class ClosedInt(int, metaclass=ClosedMeta):
    pass

This fails on some cornercases such as property and methods recovered through __getattribute__. It also fails when the base is not composed only of base types.

For example, this fails:

class MyInt(int):
    pass

class ClosedInt(MyInt, metaclass=ClosedMeta):
    pass

ClosedInt(1) + ClosedInt(1) # returns the int 2

I attempted to fix this, but it just seems to go deeper and deeper in the rabbit hole.

This seems like a problem that might have some simple pythonic solution. What would be a other, neater ways to achieve such a closed type?

Answer 1

I think using a class decorator with a black list of methods that should not return objects of the same type would be somewhat more Pythonic:

class containerize:
    def __call__(self, obj):
        if isinstance(obj, type):
            return self.decorate_class(obj)
        return self.decorate_callable(obj)

    def decorate_class(self, cls):
        for name in dir(cls):
            attr = getattr(cls, name)
            if callable(attr) and name not in ('__class__', '__init__', '__new__', '__str__', '__repr__', '__getattribute__'):
                setattr(cls, name, self.decorate_callable(attr))
        return cls

    def decorate_callable(self, func):
        def wrapper(obj, *args, **kwargs):
            return obj.__class__(func(obj, *args, **kwargs))
        return wrapper

so that:

class MyInt(int):
    pass

@containerize()
class ClosedIntContainer(MyInt):
    pass

i = ClosedIntContainer(3) + ClosedIntContainer(2)
print(i, type(i).__name__)

would output:

5 ClosedIntContainer

and that as a bonus the decorator can be selectively used on individual methods as well:

class MyInt(int):
    @containerize()
    def __add__(self, other):
        return super().__add__(other)

i = MyInt(3) + MyInt(2)
print(i, type(i).__name__)

This outputs:

5 MyInt

Answer 2

I think that the idea of using a metaclass is the way to go. The trick is to cast the values dynamically when you get them instead of up front. That's basically what python is all about: not knowing quite what you'll get or what's there until you actually get it.

To do that, you have to redefine __getattribute__ and __getattr__ on your class with some caveats:

1. Operators don't go through the normal attribute access methods. Even defining the right __getattribute__ and __getattr__ on your metaclass won't help. Dunders have to be overridden explicitly for each class.
2. Methods returned by __getattribute__ and __getattr__ need to have their return values cast to the target type. Same applies to dunders called as operators.
3. Some methods should be excepted from #2 to ensure proper operation of the machinery.

The same basic casting wrapper can be used for all the attribute and method return values. It just needs to recurse exactly oncewhen it's called on the result of __getattribute__ or __getattr__.

The solution shown below does exactly that. It explicitly wraps all dunders that aren't listed as exceptions. All other attributes are either cast immediately or wrapped if they are functions. It allows any method to be customized by checking everything in the __mro__, including the class itself. The solution will work correctly with class and static methods because it stores the casting routine and doesn't rely on type(self) (as some of my previous attempts did). It will correctly exclude any attributes listed in exceptions, not just dunder methods.

import functools


def isdunder(x):
    return isinstance(x, str) and x.startswith('__') and x.endswith('__')


class DunderSet:
    def __contains__(self, x):
        return isdunder(x)


def wrap_method(method, xtype, cast):

    @functools.wraps(method)
    def retval(*args, **kwargs):
        result = method(*args, **kwargs)
        return cast(result) if type(result) == xtype else result

    return retval


def wrap_getter(method, xtype, cast, exceptions):
    @functools.wraps(method)
    def retval(self, name, *args, **kwargs):
        result = method(self, name, *args, **kwargs)
        return result if name in exceptions else check_type(result, xtype, cast)

    return retval


def check_type(value, xtype, cast):
    if type(value) == xtype:
        return cast(value)
    if callable(value):
        return wrap_method(value, xtype, cast)
    return value


class ClosedMeta(type):
    def __new__(meta, name, bases, dct, **kwargs):
        if 'exceptions' in kwargs:
            exceptions = set([
                '__new__', '__init__', '__del__',
                '__init_subclass__', '__instancecheck__', '__subclasscheck__',
                *map(str, kwargs.pop('exceptions'))
            ])
        else:
            exceptions = DunderSet()
        target = kwargs.pop('target', bases[0] if bases else object)

        cls = super().__new__(meta, name, bases, dct, **kwargs)

        for base in cls.__mro__:
            for name, item in base.__dict__.items():
                if isdunder(name) and (base is cls or name not in dct) and callable(item):
                    if name in ('__getattribute__', '__getattr__'):
                        setattr(cls, name, wrap_getter(item, target, cls, exceptions))
                    elif name not in exceptions:
                        setattr(cls, name, wrap_method(item, target, cls))
        return cls

    def __init__(cls, *args, **kwargs):
        return super().__init__(*args)


class MyInt(int):
    def __contains__(self, x):
        return x == self
    def my_op(self, other):
        return int(self * self // other)


class ClosedInt(MyInt, metaclass=ClosedMeta, target=int,
                exceptions=['__index__', '__int__', '__trunc__', '__hash__']):
    pass

class MyClass(ClosedInt, metaclass=type):
    def __add__(self, other):
        return 1

print(type(MyInt(1) + MyInt(2)))
print(0 in MyInt(0), 1 in MyInt(0))
print(type(MyInt(4).my_op(16)))

print(type(ClosedInt(1) + ClosedInt(2)))
print(0 in ClosedInt(0), 1 in ClosedInt(0))
print(type(ClosedInt(4).my_op(16)))

print(type(MyClass(1) + ClosedInt(2)))

The result is

<class 'int'>
True False
<class 'int'> 

<class '__main__.ClosedInt'>
True False
<class '__main__.ClosedInt'>

<class 'int'>

The last example is a tribute to @wim's answer. It shows that you have to want to do this for it to work.

IDEOne link because I don't have access to a computer right now: https://ideone.com/iTBFW3

Appendix 1: Improved default exceptions

I think that a better default set of exceptions than all dunder methods can be complied by looking carefully through the special method names section of the documentation. Methods can be categorized into two broad classes: methods with very specific return types that make the python machinery work, and methods whose output should be checked and wrapped when they return an instance of your type of interest. There is a third category, which is methods that should always be excepted, even when you forget to mention them explicitly.

Here is a list of the methods that are always excepted:

• __new__
• __init__
• __del__
• __init_subclass__
• __instancecheck__
• __subclasscheck__

Here is a list of everything that should be excepted by default:

• __repr__
• __str__
• __bytes__
• __format__
• __lt__
• __le__
• __eq__
• __ne__
• __gt__
• __ge__
• __hash__
• __bool__
• __setattr__
• __delattr__
• __dir__
• __set__
• __delete__
• __set_name__
• __slots__ (not a method, but still)
• __len__
• __length_hint__
• __setitem__
• __delitem__
• __iter__
• __reversed__
• __contains__
• __complex__
• __int__
• __float__
• __index__
• __enter__
• __exit__
• __await__
• __aiter__
• __anext__
• __aenter__
• __aexit__

If we stash this list into a variable called default_exceptions, the class DunderSet can be removed entirely, and the conditional that extracts exceptions can be replaced by:

exceptions = set([
    '__new__', '__init__', '__del__',
    '__init_subclass__', '__instancecheck__', '__subclasscheck__',
    *map(str, kwargs.pop('exceptions', default_exceptions))
])

Appendix 2: Improved targetting

It should be possible to target multiple types pretty easily. This is especially useful when extending other instances of ClosedMeta, which may not override all the methods we want.

The first step in doing this is making target into a container of classes instead of a single class reference. Instead of

target = kwargs.pop('target', bases[0] if bases else object)

do

target = kwargs.pop('target', bases[:1] if bases else [object])
try:
    target = set(target)
except TypeError:
    target = {target}

Now replace every occurrence of blah == target (or blah == xtype in the wrappers) with blah in target (or blah in xtype).

Answer 3

This cannot be done, the data model forbids it. And I can prove it to you:

>>> class MyClass(ClosedInt, metaclass=type):
...     def __add__(self, other):
...         return 'potato'
...     
>>> MyClass(1) + ClosedInt(2)
'potato'

Addition is handled by the left hand object first, and if the left type handles it (i.e. does not return NotImplemented singleton) then nothing about other is considered in this operation. If the right hand type is a subclass of the left-hand type, you could control the result with the reflected method __radd__ - but of course that is impossible in the general case.