Multiprocessing in Python - process does not stop after pool.join()

Question

I have written a simple piece of code to speed up my function f, an example using 2 CPUs is shown below:

if __name__ == '__main__':
    pool = Pool(processes = 2)   
    f0_list = pool.map(f,range(nlocs))
    pool.terminate()
    pool.join()
    final_f0 = np.sum(f0_list,axis=0)

f0_final = final_f0[0:121] # here I get the error message: 'final_f0 not defined'

My result 'final_f0' is correct. However, I was expecting that, after pool.join(), all processes would end which does not seem to be the case, because:

1. I still have one CPU running in the background and it does not seem to stop even though final_f0 has been effectively returned?
2. When I try to cut 'final_f0' as described above, I get this error message. The error message prints many times as if Python was running the script over and over again.

Any suggestions?

Answer 1

You're getting the error about f0_final because you didn't keep that line inside the if __name__ == "__main__": guard. On Windows, multiprocessing needs to re-import your __main__ module in the child processes it spawns, which means everything at the top level of your module will get executed in both the parent process and all of its children.

The line

f0_final = final_f0[0:121]

is defined at the top level of the module, so it gets executed in all your child processes, however, the code under the if __name__ == "__main__": guard does not, so your attempt to take a slice of final_f0 will fail, which means all the child processes will fail. This will keep your main process from getting the results of the map call back, and prevent the pool from being properly shut down.