Cannot pickle lambda function in python 3

Question

Using dill pickling of lambda functions work fine in Python 2, but not in Python 3, is there any alternative?

Python 3:

import dill 
import pickle
pickle.dumps(lambda x: x**2)

pickle.PicklingError: Can't pickle at 0x104e97840>: attribute lookup on main failed

Python 2.7:

import dill
import pickle
pickle.dumps(lambda x: x**2)

cdill.dill\n_create_function\np0\n(cdill.dill\n_load_type\np1\n(S'CodeType'\np2\ntp3\nRp4\n(I1\nI1\nI2\nI67\nS'|\x00\x00d\x01\x00\x13S'\np5\n(NI2\ntp6\n(t(S'x'\np7\ntp8\nS''\np9\nS''\np10\nI1\nS''\np11\n(t(ttp12\nRp13\nc__main__\n__dict__\ng10\nNN(dp14\ntp15\nRp16\n.

Answer 1

I'm the dill author.

You can use dill in python3 without using dill directly... however, it's not as nice as in python2 just yet.

>>> import dill
>>> import pickle
>>> pickle._dumps(lambda x:x*x)
b'\x80\x03cdill.dill\n_create_function\nq\x00(cdill.dill\n_load_type\nq\x01X\x08\x00\x00\x00CodeTypeq\x02\x85q\x03Rq\x04(K\x01K\x00K\x01K\x02KCC\x08|\x00\x00|\x00\x00\x14Sq\x05N\x85q\x06)X\x01\x00\x00\x00xq\x07\x85q\x08X\x07\x00\x00\x00<stdin>q\tX\x08\x00\x00\x00<lambda>q\nK\x01C\x00q\x0b))tq\x0cRq\rc__main__\n__dict__\nh\nNN}q\x0etq\x0fRq\x10.'

You'll note the _dumps. Maybe you think that's weird. It is. It's because in python3, pickle has been merged with module that used to be called cPickle. Oddly:

1. import _pickle gives you the old cPickle module
2. import pickle gives you the pickle module, with cPickle merged in
3. pickle.dumps is just _pickle.dumps (i.e. it's cPickle)
4. pickle._dumps is the old pickle.dumps function

Confusing? The crux of it is: pickle.dumps is coded in C. The 2.x version of pickle.dumps was coded in python, but now it's been replaced by what was cPickle.dumps. If you want to get to the "old" version, you can... it's pickle._dumps.

When you import dill, dill automatically registers all objects it knows how to serialize to the the pickle serialization table -- the python one, not the C one. So, in python3, that means the one connected to pickle._dumps.

I suggest using dill.dumps directly instead.

>>> dill.dumps(lambda x:x*x)
b'\x80\x03cdill.dill\n_create_function\nq\x00(cdill.dill\n_load_type\nq\x01X\x08\x00\x00\x00CodeTypeq\x02\x85q\x03Rq\x04(K\x01K\x00K\x01K\x02KCC\x08|\x00\x00|\x00\x00\x14Sq\x05N\x85q\x06)X\x01\x00\x00\x00xq\x07\x85q\x08X\x07\x00\x00\x00<stdin>q\tX\x08\x00\x00\x00<lambda>q\nK\x01C\x00q\x0b))tq\x0cRq\rc__builtin__\n__main__\nh\nNN}q\x0etq\x0fRq\x10.'

I'd like to try to get the C table to work eventually...

Answer 2

Seems like in python 2, dill replaces pickle when you import. In python 3, you have to use dill directly instead.

This works in python 3.5:

>>> import dill 
>>> dill.dumps(lambda x: x**2)
b'\x80\x03cdill.dill\n_create_function\nq\x00(cdill.dill\n_load_type\nq\x01X\x08\x00\x00\x00CodeTypeq\x02\x85q\x03Rq\x04(K\x01K\x00K\x01K\x02KCC\x08|\x00\x00d\x01\x00\x13Sq\x05NK\x02\x86q\x06)X\x01\x00\x00\x00xq\x07\x85q\x08X\x07\x00\x00\x00<stdin>q\tX\x08\x00\x00\x00<lambda>q\nK\x01C\x00q\x0b))tq\x0cRq\rc__builtin__\n__main__\nh\nNN}q\x0etq\x0fRq\x10.'

Alternatively you can also import dill as pickle

>>> import dill as pickle 
>>> pickle.dumps(lambda x: x**2)