How to overcome joblib's "TypeError: can't pickle instancemethod objects" on class method?

Question

Given a class and 2 methods in it A, B.

I am getting a "TypeError: can't pickle instancemethod objects" when trying to execute the following command while in A:

joblib.Parallel(n_jobs=-1)(joblib.delayed(self.B)(arg1, arg2, arg3) for arg1 in arg1_values_list)

I found several threads on Stackoverflow regarding instancemethod and part of them are even related to joblib, but I still didn't manage to overcome this error.

EDIT:

The following code:

from sklearn.externals.joblib import Parallel, delayed
def a(self, x):
    print x
    return x*10

class c(object):
    def b(self):
        Parallel(n_jobs=-1)(
            delayed(a)(self, i) 
            for i in range(10))

test_c = c()

test_c.b()

runs ok when normally ran.

but fails when trying to profile using "python -m cProfile"

Answer 1

It is probably safer to only pass Python stateless functions to joblib.Parallel. You can try to import dill to register pickling support for instance methods: https://pypi.python.org/pypi/dill (your mileage may vary though).

Answer 2

Try giving backend="threading", multiprocess is by default and it seems to be crashing. This solved the issue for me