Python: pickling nested functions

Question

Using the example

def foo(a):
    def bar(b):
        return a+b
    return bar

d = {1:foo(1), 2:foo(2)}

It appears that pickle module will not work with a function not defined at the module scope, so pickling 'd' will not work. Is there another pickling mechanism available that I should consider?

Answer 1

I'm afraid that you can't pickle nested functions.

The pickle module serializes functions by name. That is, if you have a function myfunc in a module mymodule it simply saves the name mymodule.myfunc and looks it up again when unserializing. (This is an important security and compatibility issue, as it guarantees that the unserializing code uses its own definition for the function, rather than the original definition which might be compromised or obsolete.)

Alas, pickle can't do that with nested functions, because there's no way to directly address them by name. Your bar function, for instance, can't be accessed from outside of foo.

If you need a serializable object that works like a function, you can instead make a class with a __call__ method:

class foo(object):
    def __init__(self, a):
        self.a = a
    def __call__(self, b): # the function formerly known as "bar"
        return self.a + b

This works just like the nested functions in the question, and should pose no problem to pickle. Do be aware though, that you'll need to have the same class definition available when you unserialize a foo instance.

Answer 2

You can pickle nested functions if you use dill instead of pickle.

>>> import dill
>>>    
>>> def foo(a):
...   def bar(b):
...     return a+b
...   return bar
... 
>>> d = {1:foo(1), 2:foo(2)}
>>> 
>>> _d = dill.dumps(d)
>>> d_ = dill.loads(_d)
>>> d_
{1: <function bar at 0x108cfe848>, 2: <function bar at 0x108cfe8c0>}
>>> d[1](0) + d[2](10)
13
>>>