How to convert `lambda` object to `function` object for pickling in Python?

Question

I've been dealing with errors regarding lambda functions and their inability to be pickled. I often use lambda functions on the fly as one time use functions and it vastly decreases my workflow productivity when I have to individually recreate simple lambda functions in functional form for use with pickling.

Is there a way to convert a lambda and all of its arguments into a function object in Python 3.6.1?

lambda_func = lambda x: x.split(" ")
def func(x):
    return x.split(" ")
input_string = "Darth Plagueis was a Dark Lord of the Sith"

# Function Version
func(input_string)
# ['Darth', 'Plagueis', 'was', 'a', 'Dark', 'Lord', 'of', 'the', 'Sith']
lambda_func(input_string)
# ['Darth', 'Plagueis', 'was', 'a', 'Dark', 'Lord', 'of', 'the', 'Sith']

def lambda2func(lambda_func):
    #...
    return func_version

Answer 1

pickle does not save the code object, just its name. However, giving a function a name is not enough to make it pickleable, because unpickle deserializes it by importing the name. Naturally, this means it must be importable by the unpickling runtime. Functions defed with names can still fail to pickle for this reason. So even if you could convert your lambdas to named functions, that still wouldn't work:

>>> import pickle
>>> def foo():
...   def bar():
...     pass
...   return bar
...
>>> pickle.dumps(foo())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: Can't pickle local object 'foo.<locals>.bar'

You say your reason for not using dill instead is

I've seen a use case where dill can be imported as pickle which is pretty cool but gets confusing in the code since I used pickle and dill for different types of serialization.

But the pickle serialization format is extensible to any internal format you please —json, gzip, even dill. You can have your cake and eat it too!

Let's try it with dill, because that makes it easy:

import pickle
import dill
​
class DillPickle:
    def __init__(self, e):
        self.e = e
    def __getstate__(self):
        return dill.dumps(self.e)
    def __setstate__(self, state):
        self.e = dill.loads(state)

Pickle normally serializes the state of custom class instances by recursively pickling the object's __dict__. On unpickling, it can then make an uninitialized instance and update its __dict__ with the saved values. If all of its contents are pickleable all the way down, this works. But if not, (as in the case of the lambdas) pickling would fail. But the pickle module provides an interface to override this behavior: __getstate__() for serializing the object's state, and __setstate__() for restoring it from that value. If the return value of __getstate__() is picklable (and __setstate__() is implemented), this works. In this case, we return the picklable bytestring returned by dill.dumps().

Demonstration:

>>> pickle.dumps(DillPickle(lambda x, y: x+y))
b'\x80\x03c__main__\nDillPickle\nq\x00)\x81q\x01C\xe7\x80\x03cdill._dill\n_create_function\nq\x00(cdill._dill\n_load_type\nq\x01X\x08\x00\x00\x00CodeTypeq\x02\x85q\x03Rq\x04(K\x02K\x00K\x02K\x02KCC\x08|\x00|\x01\x17\x00S\x00q\x05N\x85q\x06)X\x01\x00\x00\x00xq\x07X\x01\x00\x00\x00yq\x08\x86q\tX\x1f\x00\x00\x00<ipython-input-18-48b9de2c6f55>q\nX\x08\x00\x00\x00<lambda>q\x0bK\x01C\x00q\x0c))tq\rRq\x0ec__builtin__\n__main__\nh\x0bNN}q\x0fNtq\x10Rq\x11.q\x02b.'
>>> pickle.loads(_).e('foo', 'bar')
'foobar'

Notice that we loaded it with pickle, not dill! And the lambda does not have a name. Of course, dill must be available to the runtime in order to unpickle, but so does the rest of the DillPickle class, just like for any custom type serialized with pickle. dill is now a mere implementation detail used internally by DillPickle. The interface is pickle.

You could even make it a callable, so you don't need to add the .e yourself.

class DillPickleCallable(DillPickle):
    def __call__(self, *args, **kwargs):
        return self.e(*args, **kwargs)