Unable to call function defined by partialmethod

Question

I'm want to use Python3 partialmethod and I can't call the resultant function. My code is based on the example from the Python documentation https://docs.python.org/release/3.6.4/library/functools.html

from functools import partialmethod
class AClass():
    def __init__(self):
        self.v = val
    def _fun(self, x):
        z = x + self.v # some computation
        return z
    def fun(self, x):
        return partialmethod(self._fun, x)

a = AClass(10)
b = a.fun(1)
print(b()) # TypeError: 'partialmethod' object is not callable
print(a.b()) # AttributeError: 'AClass' object has no attribute 'b'

I understand why a.b() is not correct, since b is not defined in the instantiated a. The error message for b() does not give me enough information to understand what is going on.

How do I define a method with bound parameters and call it from outside the enclosing class? Is this possible at all, or is there a better way to achieve this result?

Answer 1

I stumbled upon the same issue, and found your question. Reading about partialmethod more carefully, I remembered about descriptors. That is, remembered I don't really remember about descriptors. So, upon digging a bit further, here's my understanding of what is happening:

Background

Callables

Python functions and methods are, like anything else in python, objects. They are callable because they implement the callable protocol, that is, they implement the special method __call__(). You can make anything a callable this way:

class Igors:
    pass


igor = Igors()
try:
    igor()  # --> raises TypeError
except TypeError:
    print("Is it deaf?")

Igors.__call__ = lambda self: print("Did you call, Marster?")

igor()  # --> prints successfully

Prints:

Is it deaf?
Did you call, Marster?

(Note that you can't assign special methods on an instance, just on the class: Special method lookup)

Of course you'd normally do, rather:

class A:
    def __call__(self):
        print("An A was called")

a = A()
a()  # prints "An A was called"

Descriptors

Some useful links, but there are lots others:

• Descriptor HowTo Guide
• https://pabloariasal.github.io/2018/11/25/python-descriptors/

Python descriptors are objects implementing one of the __get__, __set__ or __delete__ methods. They are short-cutting the default attribute lookup mechanisms.

If an object has a "normal" attribute x, when accessing obj.x python looks for x's value in the usual suspects: the instance's __dict__, the instance's class' __dict__, then in its base classes, and returns it.

If on the other hand an object has an attribute that is a descriptor, after looking it up, python will call the descriptor's __get__() with two arguments: the instance and the instance's type (the class).

Note: the discussion is more complicated. See the linked "Descriptor HowTo Guide" for more details about __set__ and __delete__ and data vs. "non-data" descriptors and the order of looking them up.

Here's another silly example:

class Quack:
    DEFAULT = "Quack! Quack!"

    def __get__(self, obj, obj_type=None):
        print(f">>> Quacks like {obj} of type {obj_type} <<<")
        try:
            return obj.QUACK
        except AttributeError:
            return Quack.DEFAULT


class Look:
    def __get__(self, obj, obj_type):
        print(f">>> Looks like {obj} <<<")
        return lambda: "a duck!"


class Duck:
    quack = Quack()
    look = Look()


class Scaup(Duck):
    """I'm a kind of a duck"""
    QUACK = "Scaup! Scaup!"


# looking up on the class
print(f"All ducks quack: {Duck.quack}\n")

# looking up on an object
a_duck = Duck()
print(f"A duck quacks like {a_duck.quack}\n")

a_scaup = Scaup()
print(f"A scaup quacks like {a_scaup.quack}\n")

# descriptor returns a callable
print(f"A duck look like {a_duck.look} ... ooops\n")
print(f"Again, a duck look() like {a_duck.look()}\n")

Which prints:

>>> Quacks like None of type <class '__main__.Duck'> <<<
All ducks quack: Quack! Quack!

>>> Quacks like <__main__.Duck object at 0x103d5bd50> of type <class '__main__.Duck'> <<<
A duck quacks like Quack! Quack!

>>> Quacks like <__main__.Scaup object at 0x103d5bc90> of type <class '__main__.Scaup'> <<<
A scaup quacks like Scaup! Scaup!

>>> Looks like <__main__.Duck object at 0x103d5bd50> <<<
A duck look like <function Look.__get__.<locals>.<lambda> at 0x103d52dd0> ... ooops

>>> Looks like <__main__.Duck object at 0x103d5bd50> <<<
Again, a duck look() like a duck!

What you need to remember is that the magic of calling the descriptor's special methods (__get__() in this case) happens when python looks up the attribute for an obj.attribute lookup.

When running a_duck.look() python (ok, the object.__getattribute__() mechanism) is looking up "look" more or less as usual, obtains the value which is a descriptor (a class Look instance), magically calls it's __get__()

partialmethod

partialmethod() returns a descriptor which is not a callable. Instead, its __get__() method will return the callable, in this case an appropriate functools.partial() object. The partialmethod is supposed to be, similar to a method, classmethod or staticmethod, an attribute of an object.

Here are some ways to use partialmethod. Notice its behavior is different depending on whether you call it on a descriptor (like a method, classmethod, etc) or a non-descriptor callable. From its documentation:

When func is a descriptor (such as a normal Python function, classmethod(), staticmethod(), abstractmethod() or another instance of partialmethod), calls to __get__ are delegated to the underlying descriptor, and an appropriate partial object returned as the result.

When func is a non-descriptor callable, an appropriate bound method is created dynamically. This behaves like a normal Python function when used as a method: the self argument will be inserted as the first positional argument, even before the args and keywords supplied to the partialmethod constructor.

from functools import partialmethod


class Counter:
    def __init__(self, initial):
        self._value = 0

    def __str__(self):
        return str(self._value)

    def increase(self, by):
        self._value += by

    # on descriptor (a method is a descriptor too, that is doing the "self" magic)
    increment = partialmethod(increase, 1)

    # on non-descriptor
    name = lambda self: f"Counter of {self}"
    increment2 = partialmethod(name)

    # partialmethod generator

    def increment_returner(self, by):
        return partialmethod(Counter.increase, by)


# partialmethod used as intended on methods:

c = Counter(0)
c.increment()
print(f"incremented counter: {c}")    # --> 1
print(f"c.increment: {c.increment}")  # --> functools.partial(<bound method Counter.increase of <__main__.Counter object at 0x108fa0610>>, 1)
print(f"c.increment has __call__: {hasattr(c.increment, '__call__')}")  # --> True

print()

# partialmethod used (as intended?), on non-descriptor callables
print(f"c.name() returns: {c.name()}")  # --> "Counter of 1"
print(f"c.name is: {c.name}")  # --> <bound method Counter.<lambda> of <__main__.Counter object at 0x10208dc10>>

print()

# a "partialmethod" generator

incrementer = c.increment_returner(2)
print(f"icrementer: {incrementer}")  # --> functools.partialmethod(<bound method Counter.increase of <__main__.Counter object at 0x104e74790>>, 2, )
print(f"incrementer has __call__: {hasattr(incrementer, '__call__')}")  # --> False
print(f"incrementer has __get__: {hasattr(incrementer, '__get__')}")  # --> True

incrementer.__get__(c, Counter)()
print(f"counter after 'simulating' python's magic: {c}")  # --> 3
print(f"'simulated' invocation of attribute lookup: {incrementer.__get__(c, Counter)}")  # --> functools.partial(<bound method Counter.increase of <__main__.Counter object at 0x10d7b7c50>>, 2)

And the output:

incremented counter: 1
c.increment: functools.partial(<bound method Counter.increase of <__main__.Counter object at 0x101fffb10>>, 1)
c.increment has __call__: True

c.name() returns: Counter of 1
c.name is: <bound method Counter.<lambda> of <__main__.Counter object at 0x101fffb10>>

icrementer: functools.partialmethod(<function Counter.increase at 0x102008050>, 2, )
incrementer has __call__: False
incrementer has __get__: True
counter after 'simulating' python's magic: 3
'simulated' invocation of attribute lookup: functools.partial(<bound method Counter.increase of <__main__.Counter object at 0x101fffb10>>, 2)

Answers

In your example, b() doesn't work because:

1. partialmethod returns a descriptor whose __get__() would return a properly crafted callable, a partial object working like a bound method ("injecting" self).
2. Even if you'd call b.__get__(a, AClass)() this would fail because self._fun is already a bound to self and so you get TypeError: _fun() takes 2 positional arguments but 3 were given. If I'm not mistaken, self is injected twice.

As I understand your question, you want to be able to generate methods with bound parameters. I guess you could do something like:

from functools import partial, partialmethod


class AClass():
    def __init__(self, val):
        self.v = val

    def _fun(self, x):
        z = x + self.v  # some computation
        return z

    def fun1(self, x):
        def bound_fun_caller():
            return self._fun(x)
        return bound_fun_caller

    def fun2(self, x):
        # quite silly, but here it is
        return partialmethod(AClass._fun, x).__get__(self, AClass)

    def fun3(self, x):
        return partial(AClass._fun, self, x)

    # for completeness, binding to a known value
    plus_four = partialmethod(_fun, 4)

    def add_fun(self, name, x):
        # Careful, this might hurt a lot...
        setattr(AClass, name, partialmethod(AClass._fun, x))


a = AClass(10)

b1 = a.fun1(1)
print(b1())

b2 = a.fun2(2)
print(b2())

b3 = a.fun3(3)
print(b3())

print(a.plus_four())

a.add_fun("b5", 5)
print(a.b5())