dill/pickle issue with built-in function

Question

When using dill to serialize a class:

import dill, pickle

class project(object):
    def __init__(self, name='', folder='', user_id='', version=-1 ):
        self.name, self.folder, self.user_id, self.version = name, folder, user_id, version
# Other code, not shown, since Big class .......

dill.dump(proj1, open('test_dill.pkl', 'w'), byref=False )

I get this message :

PicklingError: Can't pickle <built-in function raw_input>: it's not the same object as __builtin__.raw_input


  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dill\dill.py", line 236, in dump
    pik.dump(obj)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 224, in dump
    self.save(obj)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 401, in save_reduce
    save(args)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 554, in save_tuple
    save(element)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 401, in save_reduce
    save(args)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 554, in save_tuple
    save(element)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 401, in save_reduce
    save(args)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 554, in save_tuple
    save(element)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 401, in save_reduce
    save(args)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 554, in save_tuple
    save(element)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 425, in save_reduce
    save(state)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self

  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dill\dill.py", line 835, in save_module_dict
    StockPickler.save_dict(pickler, obj)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 687, in _batch_setitems
    save(v)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 425, in save_reduce
    save(state)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self

  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dill\dill.py", line 835, in save_module_dict
    StockPickler.save_dict(pickler, obj)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 687, in _batch_setitems
    save(v)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 425, in save_reduce
    save(state)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self

  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dill\dill.py", line 835, in save_module_dict
    StockPickler.save_dict(pickler, obj)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 687, in _batch_setitems
    save(v)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 425, in save_reduce
    save(state)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self

  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dill\dill.py", line 835, in save_module_dict
    StockPickler.save_dict(pickler, obj)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 687, in _batch_setitems
    save(v)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self

  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dill\dill.py", line 835, in save_module_dict
    StockPickler.save_dict(pickler, obj)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 687, in _batch_setitems
    save(v)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 425, in save_reduce
    save(state)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self

  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dill\dill.py", line 835, in save_module_dict
    StockPickler.save_dict(pickler, obj)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 687, in _batch_setitems
    save(v)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 606, in save_list
    self._batch_appends(iter(obj))

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 642, in _batch_appends
    save(tmp[0])

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 425, in save_reduce
    save(state)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self

  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dill\dill.py", line 835, in save_module_dict
    StockPickler.save_dict(pickler, obj)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 687, in _batch_setitems
    save(v)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 425, in save_reduce
    save(state)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self

  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dill\dill.py", line 835, in save_module_dict
    StockPickler.save_dict(pickler, obj)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 687, in _batch_setitems
    save(v)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 425, in save_reduce
    save(state)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self

  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dill\dill.py", line 835, in save_module_dict
    StockPickler.save_dict(pickler, obj)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 687, in _batch_setitems
    save(v)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self

  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dill\dill.py", line 835, in save_module_dict
    StockPickler.save_dict(pickler, obj)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 687, in _batch_setitems
    save(v)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 425, in save_reduce
    save(state)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self

  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dill\dill.py", line 835, in save_module_dict
    StockPickler.save_dict(pickler, obj)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 687, in _batch_setitems
    save(v)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self

  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dill\dill.py", line 835, in save_module_dict
    StockPickler.save_dict(pickler, obj)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 687, in _batch_setitems
    save(v)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 425, in save_reduce
    save(state)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self

  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dill\dill.py", line 835, in save_module_dict
    StockPickler.save_dict(pickler, obj)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 687, in _batch_setitems
    save(v)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self

  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dill\dill.py", line 835, in save_module_dict
    StockPickler.save_dict(pickler, obj)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 687, in _batch_setitems
    save(v)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 425, in save_reduce
    save(state)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self

  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dill\dill.py", line 835, in save_module_dict
    StockPickler.save_dict(pickler, obj)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 687, in _batch_setitems
    save(v)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self

  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dill\dill.py", line 993, in save_builtin_method
    StockPickler.save_global(pickler, obj)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 759, in save_global
    (obj, module, name))

PicklingError: Can't pickle <built-in function raw_input>: it's not the same object as __builtin__.raw_input

Am not sure than built-in functions can be modified by the code...
It sounds mysterious.

Answer 1

I'm the dill author. The different settings (such as byref) are used to modify the types of objects that are included in the dump. pickle, and thus dill, use recursive object serialization, which means when an object refers to another object, it may (or may not) serialize it, based on the serialization settings. pickle typically serializes functions and classes by reference (byref=True) which means they are only serialized on import and by name. dill has several settings, such as byref=False (which serializes function and class objects instead of by reference). There is also recurse, which deals with how global references are serialized -- recurse=False serializes all the globals as a dict, while recurse=True serializes only the referenced items in globals and items that those items refer to and so on.

In short, whatever you are serializing has some global references, which trigger recursive serialization until it hits something dill can't pickle.

Here are some things you can try:

1. Separate objects in the file into separate files -- this allows you to rely more on pickling by reference.
2. Use different settings, such as recurse=True, to change which objects dill tries to serialize with your target object. You can set the settings globally with dill.settings.
3. Add serialization helper methods to your class, that inform pickle how to serialize your class.
4. Refactor your class (somehow, since I can't see the contents of it) to avoid serialization problems.

It's hard to answer more in depth as you haven't posted a working example that demonstrates the error.