If I have a reference to a bound method in Python, will that alone keep the object alive?

Question

I wrote something like this today (not unlike the mpl_connect documentation:

class Foo(object):
    def __init__(self): print 'init Foo', self
    def __del__(self): print 'del Foo', self
    def callback(self, event=None): print 'Foo.callback', self, event


from pylab import *
fig = figure()
plot(randn(10))
cid = fig.canvas.mpl_connect('button_press_event', Foo().callback)
show()

This looks reasonable, but it doesn't work -- it's as though matplotlib loses track of the function I've given it. If instead of passing it Foo().callback I pass it lambda e: Foo().callback(e), it works. Similarly if I say x = Foo() and then pass it x.callback, it works.

My presumption is that the unnamed Foo instance created by Foo() is immediately destroyed after the mpl_connect line -- that matplotlib having the Foo.callback reference doesn't keep the Foo alive. Is that correct?

In the non-toy code I encountered this in, the solution of x = Foo() didn't work, presumably because in that case show() was elsewhere so x had gone out of scope.

More generally, Foo().callback is a <bound method Foo.callback of <__main__.Foo object at 0x03B37890>>. My primary surprise is that it seems like a bound method isn't actually keeping a reference to the object. Is that correct?

Answer 1

Yes, a bound method references the object - the object is the value of a bound method object's .im_self attribute.

So I'm wondering whether matplotlib's mpl_connect() is remembering to increment the reference counts on the arguments passed to it. If not (and this is a common error), then there's nothing to keep the anonymous Foo().callback alive when mpl_connect() returns.

If you have easy access to the source code, take a look at the mpl_connect() implementation? You want to see C code doing Py_INCREF() ;-)

EDIT This looks relevant, from docs here:

The canvas retains only weak references to the callbacks. Therefore if a callback is a method of a class instance, you need to retain a reference to that instance. Otherwise the instance will be garbage- collected and the callback will vanish.

So it's your fault - LOL ;-)

Answer 2

Here's the justification from matplotlib.cbook.CallbackRegistry.__doc__:

In practice, one should always disconnect all callbacks when they are no longer needed to avoid dangling references (and thus memory leaks). However, real code in matplotlib rarely does so, and due to its design, it is rather difficult to place this kind of code. To get around this, and prevent this class of memory leaks, we instead store weak references to bound methods only, so when the destination object needs to die, the CallbackRegistry won't keep it alive. The Python stdlib weakref module can not create weak references to bound methods directly, so we need to create a proxy object to handle weak references to bound methods (or regular free functions). This technique was shared by Peter Parente on his "Mindtrove" blog <http://mindtrove.info/articles/python-weak-references/>_.

It's a shame there isn't an official way to bypass this behavior.

Here's a kludge to get around it, which is dirty but may be OK for non-production test/diagnostic code: attach the Foo the the figure:

fig._dont_forget_this = Foo()
cid = fig.canvas.mpl_connect('button_press_event', fig._dont_forget_this.callback)

This still leaves the question of why lambda e: Foo().callback(e) works. It obviously makes a new Foo at every call, but why doesn't the lambda get garbage collected? Is the fact that it works just a case of undefined behavior?