Pickling weakref in Python

Question

I am still pretty new to Python and even newer to pickling. I have a class Vertex(ScatterLayout) with a __getnewargs__():

def __getnewargs__(self):
    return (self.pos, self.size, self.idea.text)

My understanding is that this will cause the pickle to pickle the object from __getnewargs__() rather than the object's dictionary.

The pickle is called in the following method (in a different class MindMapApp(App)):

def save(self):
    vertices = self.mindmap.get_vertices()
    edges = self.mindmap.get_edges()

    output = open('mindmap.pkl', 'wb')

    #pickle.dump(edges, output, pickle.HIGHEST_PROTOCOL)
    pickle.dump(vertices, output, pickle.HIGHEST_PROTOCOL)

    output.close()

When I call the save() method I get the following error:

pickle.PicklingError: Can't pickle <type 'weakref'>: it's not found as __builtin__.weakref

What am I missing or not understanding? I have also tried implementing the __getstate__() / __setstate__(state) combination, with the same result.

Answer 1

You definitely can pickle a weakref, and you can pickle a dict and a list. However, it actually matters what they contain. If the dict or list contains any unpicklable items, then the pickling will fail. If you want to pickle a weakref, you have to use dill and not pickle. The unpickled weakref however deserialize as dead references.

>>> import dill
>>> import weakref
>>> dill.loads(dill.dumps(weakref.WeakKeyDictionary()))
<WeakKeyDictionary at 4528979192>
>>> dill.loads(dill.dumps(weakref.WeakValueDictionary()))
<WeakValueDictionary at 4528976888>
>>> class _class:
...   def _method(self):
...     pass
... 
>>> _instance = _class()
>>> dill.loads(dill.dumps(weakref.ref(_instance)))
<weakref at 0x10d748940; dead>
>>> dill.loads(dill.dumps(weakref.ref(_class())))
<weakref at 0x10e246a48; dead>
>>> dill.loads(dill.dumps(weakref.proxy(_instance)))
<weakproxy at 0x10e246b50 to NoneType at 0x10d481598>
>>> dill.loads(dill.dumps(weakref.proxy(_class())))
<weakproxy at 0x10e246ba8 to NoneType at 0x10d481598>

Answer 2

I worked around this by switching between weak/strong reference in __getstate__/__setstate__:

class DBObject(object):
    def __getstate__(self):
        s = self.__dict__.copy()
        s['_db'] = s['_db']()
        return s

    def __setstate__(self, state):
        self.__dict__ = state.copy()
        self.__dict__['_db'] = weakref.ref(self.__dict__['_db'])