Dynamically inherit all Python magic methods from an instance attribute

Question

I ran into an interesting situation while working on a project:

I am constructing a class, which we can call ValueContainer which always will store a single value under a value attribute. ValueContainer to have custom functionality, keep other metadata, etc., however I would like to inherit all the magic/dunder methods (e.g. __add__, __sub__, __repr__) from value. The obvious solution is to implement all magic methods by hand and point the operation to the value attribute.

Example definition:

class ValueContainer:

    def __init__(self, value):
        self.value = value

    def __add__(self, other):
        if isinstance(other, ValueContainer):
            other = other.value
        return self.value.__add__(other)

Example behavior:

vc1 = ValueContainer(1)
assert vc1 + 2 == 3
vc2 = ValueContainer(2)
assert vc1 + vc2 == 3 

However, there are two issues here.

1. I want to inherit ALL magic methods from type(self.value), which would end up being likely 20+ different functions, all with the same core functionality (calling the super magic-method of value). This makes my every ounce of my body shiver, and shout "DRY! DRY! DRY!"
2. value can be any type. At the VERY least I need to support at least numeric types (int, float) and strings. The set of magic methods and their behaviors for numerics and strings are already different enough to make this a sticky situation to handle. Now when adding the fact that I would like the ability to store custom types in value, it becomes somewhat unimaginable to implement manually.

With these two things in mind, I spend a long time trying different approaches to get this working. The tough part comes from the fact that dunder methods are class properties(?), but value gets assigned to an instance.

Attempt 1: After value is assigned, we look up all the methods that start with __ on the class type(self.value), and assign the class dunder methods on ValueContainer to be these functions. This seemed to be a good solution at first, before realizing the doing this would now reassign the dunder methods of ValueContainer for all instances.

This means when we instantiate:

valc_int = ValueContainer(1)

it will apply all dunder methods from int to the ValueContainer class. Great!

...but if we then instantiate:

valc_str = ValueContainer('a string')

all the dunder methods for str will be set on the class ValueContainer, meaning valc_int would now try to use the dunder methods from str, potentially causing issue when there's overlap.

Attempt 2: This is the solution I am currently using, which achieves the majority of the functionality that I'm after.

Welcome, Metaclasses.

import functools

def _magic_function(valc, method_name, *args, **kwargs):
    if hasattr(valc.value, method_name):
        # Get valc.value's magic method
        func = getattr(valc.value, method_name)
        # If comparing to another ValueContainer, need to compare to its .value
        new_args = [arg.value if isinstance(arg, ValueContainer)
                    else arg for arg in args]
        return func(*new_args, **kwargs)


class ValueContainerMeta(type):
    blacklist = [
        '__new__',
        '__init__',
        '__getattribute__',
        '__getnewargs__',
        '__doc__',
    ]

    # Filter magic methods
    methods = {*int.__dict__, *str.__dict__}
    methods = filter(lambda m: m.startswith('__'), methods)
    methods = filter(lambda m: m not in ValueContainer.blacklist, methods)

    def __new__(cls, name, bases, attr):
        new = super(ValueContainer, cls).__new__(cls, name, bases, attr)

        # Set all specified magic methods to our _magic_function
        for method_name in ValueContainerMeta.methods:
            setattr(new, method_name, functools.partialmethod(_magic_function, method_name))

        return new


class ValueContainer(metaclass=ValueContainerMeta):

    def __init__(self, value):
        self.value = value

Explanation:

By using the ValueContainerMeta metaclass, we intercept the creation of ValueContainer, and override the specific magic methods that we collect on the ValueContainerMeta.methods class attribute. The magic here comes from the combination of our _magic_function function and functools.partialmethod. Just like a dunder method, _magic_function takes the ValueContainer instance it is being called on as the first parameter. We'll come back to this in a second. The next argument, method_name, is the string name of the magic method we want to call ('__add__' for example). The remaining *args and **kwargs will be the arguments that would be passed to the original magic method (generally no arguments or just other, but sometimes more).

In the ValueContainerMeta metaclass, we collect a list of magic methods to override, and use partialmethod to inject the method name to call without actually calling the _magic_function itself. Initially I though just using functools.partial would serve the purpose since dunder methods are class methods, but apparently magic methods are somehow also bound to instances even though they are class methods? I still don't fully understand the implementation, but using functools.partialmethod solves this issue by injecting the ValueContainer instance be called as the first argument in _magic_fuction (valc).

Output:

def test_magic_methods():
    v1 = ValueContainer(1.0)

    eq_(v1 + 4, 5.0)
    eq_(4 + v1, 5.0)
    eq_(v1 - 3.5, -2.5)
    eq_(3.5 - v1, 2.5)
    eq_(v1 * 10, 10)
    eq_(v1 / 10, 0.1)

    v2 = ValueContainer(2.0)

    eq_(v1 + v2, 3.0)
    eq_(v1 - v2, -1.0)
    eq_(v1 * v2, 2.0)
    eq_(v1 / v2, 0.5)

    v3 = ValueContainer(3.3325)
    eq_(round(v3), 3)
    eq_(round(v3, 2), 3.33)

    v4 = ValueContainer('magic')
    v5 = ValueContainer('-works')

    eq_(v4 + v4, 'magicmagic')
    eq_(v4 * 2, 'magicmagic')
    eq_(v4 + v5, 'magic-works')

    # Float magic methods still work even though
    # we instantiated a str ValueContainer
    eq_(v1 + v2, 3.0)
    eq_(v1 - v2, -1.0)
    eq_(v1 * v2, 2.0)
    eq_(v1 / v2, 0.5)

Overall, I am happy with this solution, EXCEPT for the fact that you must specify which method names to inherit explicitly in ValueContainerMeta. As you can see, for now I've taken the superset of str and int magic methods. If possible, I would love a way to dynamically populate the list of method names based on the type of value, but since this is happening before its instantiation, I don't believe that would be possible with this approach. If there are magic methods on a type that aren't contained within the superset of int and str right now, this solution would not work with those.

Although this solution is 95% of what I am looking for, it was such an interesting problem that I wanted to know if anyone else could come up with a better solution, that accomplishes dynamic choosing of magic methods from the type of value, or has optimizations/tricks for improving other aspects, or if someone could explain more of the internals of how magic methods work.

Answer 1

As you've correctly identified,

1. magic methods are discovered on the class, not on the instance, and
2. you have no access to the wrapped value prior to class creation.

With that in mind, I think it's impossible to force instances of the same class to overload operators differently depending on the wrapped value type.

One workaround is to dynamically create and cache ValueContainer subclasses. For example,

import inspect

blacklist = frozenset([
    '__new__',
    '__init__',
    '__getattribute__',
    '__getnewargs__',
    '__doc__',
    '__setattr__',
    '__str__',
    '__repr__',
])

# container type superclass
class ValueContainer:
    def __init__(self, value):
        self.value = value

    def __repr__(self):
        return '{}({!r})'.format(self.__class__.__name__, self.value)

# produce method wrappers
def method_factory(method_name):
    def method(self, other):
        if isinstance(other, ValueContainer):
            other = other.value
        return getattr(self.value, method_name)(other)
    return method

# create and cache container types (instances of ValueContainer)
type_container_cache = {}
def type_container(type_, blacklist=blacklist):
    try:
        return type_container_cache[type_]
    except KeyError:
        pass

    # e.g. IntContainer, StrContainer
    name = f'{type_.__name__.title()}Container'
    bases = ValueContainer,
    method_names = {
        method_name for method_name, _ in inspect.getmembers(type_, inspect.ismethoddescriptor) if
        method_name.startswith('__') and method_name not in blacklist
    }

    result = type_container_cache[type_] = type(name, bases, {
        n: method_factory(n) for n in method_names})
    return result

# create or lookup an appropriate ValueContainer
def value_container(value):
    cls = type_container(type(value))
    return cls(value)

You can then use the value_container factory.

i2 = value_container(2)
i3 = value_container(3)
assert 2 + i2 == 4 == i2 + 2
assert repr(i2) == 'IntContainer(2)'
assert type(i2) is type(i3)

s = value_container('a')
assert s + 'b' == 'ab'
assert repr(s) == "StrContainer('a')"