python multiprocessing pickle protocol

Question

I am using the Python multiprocessing module to place objects onto a queue and have them processed by several workers. My first issue was getting bound instance methods to pickle, which I have working, but now I'm running into a separate issue caused by the fact that the objects are using __slots__.

When the mp module goes to pickle the objects, it seems to be using the older ascii pickle protocol, that can't handle __slots__. The newer protocol does handle this, but I'm not sure how to make the mp module use this protocol.

Anyone have any experience with this?

Answer 1

If it's not possible to change the pickle protocol the multiprocessing package uses, then define __getstate__ and __setstate__ for your objects:

import pickle

class Foo(object):
    __slots__ = ['this', 'that', 'other']

    def __init__(self):
        self.this = 1
        self.that = 2
        self.other = 3

    def __getstate__(self):
        return dict((name, getattr(self, name))
                    for name in self.__slots__)

    def __setstate__(self, state):
        for name, value in state.items():
            setattr(self, name, value)

pickle.dumps(Foo(), protocol=0)