Error while using joblib with imported function

Question

I'm using joblib to parallelize my python 3.5 code.
If I do:

from modules import f
from joblib  import Parallel, delayed

if __name__ == '__main__':
    Parallel( n_jobs  =  n_jobs,backend = "multiprocessing")(delayed(f)(i) for i in range( 10 ))

code doesn't work. Instead:

from joblib import Parallel, delayed

def f( i ):
    # my func ...

if __name__ == '__main__':
    Parallel( n_jobs  =  n_jobs, backend = "multiprocessing")(delayed(f)(i) for i in range(10))

This works!

Can someone explain why I have to put all my functions in the same script?

That is really unpractical, because in modules there are plenty of functions that I coded, that I don't want to copy / paste in the main script.

Answer 1

I faced the similar ussue. When I call function from import, it just freezes and when I call local function it works OK. Solve it by using multithreading instead of multiprocessing like that

Parallel( n_jobs  =  n_jobs, backend='threading')(delayed(f)(i) for i in range(10))

Answer 2

I found a workaround that allows you to keep the helper functions in separates module. For each imported function that you want to parallelize, define a proxy function in your main module, e.g. as

def f_proxy(*args, **kwargs):
    return f(*args, **kwargs)

and simply use delayed(f_proxy). It is still somewhat unsatisfactory, but cleaner than moving all helper functions into the main module.