Multiprocessing Pool initializer fails pickling

Question

I am trying to use the multiprocessing.Pool to implement a multithread application. To share some variables I am using a Queue as hinted here:

def get_prediction(data):
    #here the real calculation will be performed
    ....


def mainFunction():
    def get_prediction_init(q):
        print("a")
        get_prediction.q = q

    queue = Queue()
    pool = Pool(processes=16, initializer=get_prediction_init, initargs=[queue,])

if __name__== '__main__':
    mainFunction()

This code is running perfectly on a Debian machine, but is not working at all on another Windows 10 device. It fails with the error

AttributeError: Can't pickle local object 'mainFunction.<locals>.get_prediction_init'

I do not really know what exactly is causing the error. How can I solve the problem so that I can run the code on the Windows device as well?

EDIT: The problem is solved if I create the get_predediction_init function on the same level as the mainFunction. It has only failed when I defined it as an inner function. Sorry for the confusion in my post.

Answer 1

The problem is in something you haven't shown us. For example, it's a mystery where "mainFunction" came from in the AttributeError message you showed.

Here's a complete, executable program based on the fragment you posted. Worked fine for me under Windows 10 just now, under Python 3.6.1 (I'm guessing you're using Python 3 from your print syntax), printing "a" 16 times:

import multiprocessing as mp

def get_prediction(data):
    #here the real calculation will be performed
    pass

def get_prediction_init(q):
    print("a")
    get_prediction.q = q

if __name__ == "__main__":
    queue = mp.Queue()
    pool = mp.Pool(processes=16, initializer=get_prediction_init, initargs=[queue,])
    pool.close()
    pool.join()

Edit

And, based on your edit, this program also works fine for me:

import multiprocessing as mp

def get_prediction(data):
    #here the real calculation will be performed
    pass

def get_prediction_init(q):
    print("a")
    get_prediction.q = q

def mainFunction():
    queue = mp.Queue()
    pool = mp.Pool(processes=16, initializer=get_prediction_init, initargs=[queue,])
    pool.close()
    pool.join()

if __name__ == "__main__":
    mainFunction()

Edit 2

And now you've moved the definition of get_prediction_init() into the body of mainFunction. Now I can see your error :-)

As shown, define the function at module level instead. Trying to pickle local function objects can be a nightmare. Perhaps someone wants to fight with that, but not me ;-)