The callback does not call when multiprocessing

Question

I take a whole day to debug the code below. It is about multiprocessing. Please take a look.

import numpy as np
import multiprocessing as mp


def printHello(x):
    print "ENTERING: printHello"
    time.sleep(2)
    print "QUITTING: printHello"
    return 'hello '+x

def log_result(result):
    print "ENTERING: log_result"
    time.sleep(2)
    print "QUITTING: log_result"

def main_multi():
    pool = mp.Pool()
    for i  in range(3):
        pool.apply_async(printHello, args=(i, ), callback=log_result)
    pool.close()
    pool.join()

if __name__=='__main__':
    print "ENTERING __main__"
    main_multi()
    print "QUITTING __main__"

I thought I would get "QUITTING: printHello", "ENTERING: log_result" and "QUITTING: printHello" among the output lines. What appears strange to me is that, printHello has not yet finished when the main program stops (no 'hello' printed out); the 'pool.join' statement seems be to ignored in some way. Maybe the 'callback' or 'apply_async' above do not work as I expected. Any idea?

The output from my console is:

ENTERING __main__
ENTERING: printHello
ENTERING: printHello
ENTERING: printHello
QUITTING __main__ 

[EDIT] Interestingly, if I remove the 'time.sleep(2)' part, I get

ENTERING __main__
ENTERING: printHello
QUITTING: printHello
ENTERING: printHello
QUITTING: printHello
ENTERING: printHello
QUITTING: printHello
QUITTING __main__

But this still does not solve all the problems. Because 'hello' in 'helloPrint' is not printed out, whereas I do have a pool.join after apply_async.

[Edit2] I just add the 'import time'. The symptom persists: pool.join should block until the process terminates, but that is not the case here.

Answer 1

You haven't imported the time module, so your calls to time.sleep cause the worker processes to error out. This is why you don't get the "QUITTING" messages, nor does the callback get called.

If you fix that issue, you'll still be getting errors in the workers when you try to produce the return value. You're adding the integer x you get passed to the string "hello", which is not well defined. You need to convert the integer to a string for the string concatenation to work.