Python serialize lexical closures?

Question

Is there a way to serialize a lexical closure in Python using the standard library? pickle and marshal appear not to work with lexical closures. I don't really care about the details of binary vs. string serialization, etc., it just has to work. For example:

def foo(bar, baz) :     def closure(waldo) :         return baz * waldo     return closure 

I'd like to just be able to dump instances of closure to a file and read them back.

Edit: One relatively obvious way that this could be solved is with some reflection hacks to convert lexical closures into class objects and vice-versa. One could then convert to classes, serialize, unserialize, convert back to closures. Heck, given that Python is duck typed, if you overloaded the function call operator of the class to make it look like a function, you wouldn't even really need to convert it back to a closure and the code using it wouldn't know the difference. If any Python reflection API gurus are out there, please speak up.

Answer 1

PiCloud has released an open-source (LGPL) pickler which can handle function closure and a whole lot more useful stuff. It can be used independently of their cloud computing infrastructure - it's just a normal pickler. The whole shebang is documented here, and you can download the code via 'pip install cloud'. Anyway, it does what you want. Let's demonstrate that by pickling a closure:

import pickle from StringIO import StringIO  import cloud  # generate a closure def foo(bar, baz):     def closure(waldo):         return baz * waldo     return closure closey = foo(3, 5)  # use the picloud pickler to pickle to a string f = StringIO() pickler = cloud.serialization.cloudpickle.CloudPickler(f) pickler.dump(closey)  #rewind the virtual file and reload f.seek(0) closey2 = pickle.load(f) 

Now we have closey, the original closure, and closey2, the one that has been restored from a string serialisation. Let's test 'em.

>>> closey(4) 20 >>> closey2(4) 20 

Beautiful. The module is pure python—you can open it up and easily see what makes the magic work. (The answer is a lot of code.)

Answer 2

If you simply use a class with a __call__ method to begin with, it should all work smoothly with pickle.

class foo(object):     def __init__(self, bar, baz):         self.baz = baz     def __call__(self,waldo):         return self.baz * waldo 

On the other hand, a hack which converted a closure into an instance of a new class created at runtime would not work, because of the way pickle deals with classes and instances. pickle doesn't store classes; only a module name and class name. When reading back an instance or class it tries to import the module and find the required class in it. If you used a class created on-the-fly, you're out of luck.