Why can’t I pickle an object that references a function defined inside another function?

Question

I’m trying to pickle an object that holds a reference to a function defined in a local scope. Pickle fails, even with dill or cloudpickle. What are the technical reasons for this, and is there a workaround for serializing such closures?

Answer 1

So, for a start - ordinary pickle does not serialize either functions (non-nested) or classes:

What it does serialize is the module name and reference to the function or class, and upon unpickling, it fetches back the same object in the same module: if the unpickling takes place in the same program that created the pickle in the first place, then the reference will be the same.

But if it just happens to unpickle an object which references a class (or an instance of a class, for that matter) in a module which is not the same in the unpickling environment as it was in the serialization environment, things will just break or worse.

Here is a shell session where I create a module, pickle a reference to a function in it, then change the code and unpickle it back:

(env313t) user@host:~/tmp01$ cat >test.py
def dup(x): 
    return x * 2
(env313t) user@host:~/tmp01$ python
Python 3.13.2+ (...)
>>> import test
>>> a = [test.dup]
>>> a[0](4)
8
>>> import pickle
>>> pickle.dump(a, open("test.pickle", "wb"))
>>> 
(env313t) user@host:~/tmp01$ rm test.py
(env313t) user@host:~/tmp01$ cat >test.py
def dup(x):
    return x / 2
(env313t) user@host:~/tmp01$ python
Python 3.13.2+ (...)
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle
>>> a = pickle.load(open("test.pickle", "rb")) # note 'test' wasn't imported - but test.py i\
s on the FS
>>> a[0](4)
2.0
>>> 

Now, for nested functions, there simply is not a way to get a dotted name reference (i.e. module.parent_func.inner_func()) to the inner function - so that it can be reassigned and referenced on unpickling.

You get, though, the function object itself, and it can be rebuilt to a certain point. That is what projects like dill do for module level functions and classes.

But for nested functions there is one extra factor - it will reference a "closure" object: and object which will reference the non-local variables it will use, and that is non-pickable by nature, as it depends on live, in memory objects.

I suspect that is the point dill will bail out serializing a nested function.

Although the contents of nonlocal variables can be accessed and frozen at serialization time, and these could be re-built, as separate instances on unpickling, the unserialized object would no longer be 'connected' to the same space. But then, neither other mutable objects are the same instances when unpickling - so, it could be done.

The "pickle protocol" is quite flexible as it is, and it allows registering another pickle class that could, upon finding a function reference, serialize all its components, and re-create a function upon unpickling, by using a types.FunctionType call, and recreating a proper __closure__ object to be used on that call.

So, that would be your proper workaround. However, I think it would take at least a couple hours to implement correctly (and likely more, as one'd be recreating functionality from pickle itself, a feature developed through decades), so I am not adding code to that on this answer - sorry for that.