PicklingError: Can't pickle <type 'function'> with python process pool executor

Question

util.py

def exec_multiprocessing(self, method, args):
    with concurrent.futures.ProcessPoolExecutor() as executor:
        results = pool.map(method, args)
        return results

clone.py

def clone_vm(self, name, first_run, host, ip):
    # clone stuff

invoke.py

exec_args = [(name, first_run, host, ip) for host, ip in zip(hosts, ips)]
results = self.util.exec_multiprocessing(self.clone.clone_vm, exec_args)

The above code gives the pickling error. I found that it is because we are passing instance method. So we should unwrap the instance method. But I am not able to make it work.

Note: I can not create top level method to avoid this. I have to use instance methods.

Answer 1

Let's start with an overview - why the error came up in the first place:

The multiprocessing must requires to pickle (serialize) data to pass them along processes or threads. To be specific, pool methods themselves rely on queue at the lower level, to stack tasks and pass them to threads/processes, and queue requires everything that goes through it must be pickable.

The problem is, not all items are pickable - list of pickables - and when one tries to pickle an unpicklable object, gets the PicklingError exception - exactly what happened in your case, you passed an instance method which is not picklable.

There can be various workarounds (as is the case with every problem) - the solution which worked for me is here by Dano - is to make pickle handle the methods and register it with copy_reg.

---

Add the following lines at the start of your module clone.py to make clone_vm picklable (do import copy_reg and types):

def _pickle_method(m):
    if m.im_self is None:
        return getattr, (m.im_class, m.im_func.func_name)
    else:
        return getattr, (m.im_self, m.im_func.func_name)

copy_reg.pickle(types.MethodType, _pickle_method)

---

Other useful answers - by Alex Martelli, mrule, by unutbu

Answer 2

You need to add support for pickling functions and methods for that to work as pointed out by Nabeel Ahmed. But his solution won't work with name-mangled methods -

import copy_reg
import types

def _pickle_method(method):
    attached_object = method.im_self or method.im_class
    func_name = method.im_func.func_name

    if func_name.startswith('__'):
        func_name = filter(lambda method_name: method_name.startswith('_') and method_name.endswith(func_name), dir(attached_object))[0]

    return (getattr, (attached_object, func_name))

copy_reg.pickle(types.MethodType, _pickle_method)

This would work for name mangled methods as well. For this to work, you need to ensure this code is always ran before any pickling happens. Ideal place is settings file(if you are using django) or some package that is always imported before other code is executed.

Credits:- Steven Bethard (https://bethard.cis.uab.edu/)