Python - How can I make this un-pickleable object pickleable?

Question

So, I have an object that has quite a bit of non-pickleable things in it (pygame events, orderedDicts, clock, etc.) and I need to save it to disk.

Thing is, if I can just get this thing to store a string that has the progress (a single integer is all I need), then I can pass it to the object's init and it will rebuild all of those things. Unfortunately, a framework I am using (Renpy) will pickle the object and attempt to load it, despite the fact that I could save it as a single integer, and I can't change that.

So, what I'm asking is, how can I override methods so that whenever pickle tries to save the object, it saves only the progress value, and whenever it tries to load the object, it creates a new instance from the progress value?

I've seen a bit talking bout the __repr__ method, but I am unsure how I would use this in my situation.

Answer 1

The hook you're looking for is __reduce__. It should return a (callable, args) tuple; the callable and args will be serialized, and on deserialization, the object will be recreated through callable(*args). If your class's constructor takes an int, you can implement __reduce__ as

class ComplicatedThing:
    def __reduce__(self):
        return (ComplicatedThing, (self.progress_int,))

There are a few optional extra things you can put into the tuple, mostly useful for when your object graph has cyclic dependencies, but you shouldn't need them here.

Answer 2

While using __reduce__ is a valid way to do this, as the Python docs state:

Although powerful, implementing __reduce__() directly in your classes is error prone. For this reason, class designers should use the high-level interface (i.e., __getnewargs_ex__(), __getstate__() and __setstate__()) whenever possible

So, I'll explain how to use the simpler higher-level interfaces __getstate__ and __setstate__ to make an object picklable.

Let's take a very simple class with an unpicklable attribute, let's say it's a file handle.

class Foo:
    def __init__(self, filename):
        self.filename = filename
        self.f = open(filename) # this attribute cannot be pickled

Instances of Foo are not pickable:

obj = Foo('test.txt')
pickle.dumps(obj)
# TypeError: cannot pickle '_io.TextIOWrapper' object

We can make this class serializable and deserializable using pickle by implementing __getstate__ and __setstate__, respectively.

class Foo:
    ... # the class as it was
    def __getstate__(self):
       """Used for serializing instances"""
       
       # start with a copy so we don't accidentally modify the object state
       # or cause other conflicts
       state = self.__dict__.copy()

       # remove unpicklable entries
       del state['f']
       return state

    def __setstate__(self, state):
        """Used for deserializing"""
        # restore the state which was picklable
        self.__dict__.update(state)
        
        # restore unpicklable entries
        f = open(self.filename)
        self.f = f

Now it can be pickled:

obj = Foo('text.txt')
pickle.dumps(obj)
# b'\x80\x04\x951\x00\x00\x00\x00\x00\x00\x00\x8c\x08[...]'

Applying this idea to the example in your question, you might do something like this:

class MyComplicatedObject:
    def __getstate__(self):
        state = self.__dict__.copy()
        del state['progress'] # remove the unpicklable progress attribute
        return state
    def __setstate__(self, state):
        self.__dict__.update(state)
        # restore the progress from the progress integer
        self.progress = make_progress(self.progress_int)

---

Another way to do this would be to configure the pickler to know how to pickle new objects (rather than making the classes/objects themselves picklable). For example, with a custom pickler and dispatch_table you can register classes to functions (__reduce__-like) in order to pickle objects that may otherwise not be picklable.

In Python 3.8+ you can also implement custom reductions for objects.

These methods are particularly useful if you are trying to pickle classes that may belong to third party libraries/code where subclassing (to make the object picklable) is not practical.