python script hangs on importing module with multiprocessing code

Question

I am trying to import a simple module, which uses multiprocessing into my main file. The module using multiprocessing obtains the computed results from an asyncresult.get(). When this routine is called via importing the script just hangs and does not continue.

Here is a small example.

import_test.py (the module to import)

import sys
import multiprocessing as mp

if not hasattr(sys.stdin, 'close'):
    def dummy_close():
        pass
    sys.stdin.close = dummy_close

# simple test function to compute in parallel
def testf(x):
    return x*x

# multiprocessing code
pool = mp.Pool()
results = []
for i in range(0,2):
    results.append(pool.apply_async(testf, [i]))
for i in range(0,2):
    print results[i].get()
pool.close()
pool.join()

main.py (simple script which just imports the code and should print stuff from import_test)

if __name__ == '__main__':
    import import_test

print "done"

When running the script I can see that the import takes place, but the first time results[i].get() (asyncresult.get() routine) is called the whole script hangs and does not continue without throwing any error. I have tested this under Mac OS X (El Capitan) and Windows 10 running two obviously different python setups. The result is always the behavior described above.

Everything works fine if I just put the code from the imported module into main.py itself. Of course my actual code is more complicated than that and I want to keep the parallel computation in the module to be imported.

How can I fix this problem?

Answer 1

I may be coming too late, but I think I have the answer to your problem. I had the same a while back. It stems from how Python handles GIL locks during imports and how it clashes with multiprocessing creating a deadlock.

You should have a look at https://docs.python.org/2/library/imp.html. The problem is the following: Python acquires the GIL lock when doing an import and releases it at the end. Hence, when you run a thread via multiprocessing in your imported module and it needs to acquire the lock, a deadlock happens. That is why I advise you to do multiprocessing in your main file.

If for some reason you absolutely want to do multiprocessing during an import, there is also a solution:

 import imp
 if imp.lock_held():
     imp.release_lock()
     # do your multiprocessing stuff
     imp.acquire_lock() # Don't forget this import needs to have the lock acquired at the end of the import

If you don't make imp acquire the lock anew, you-ll get a runtime exception.

I hope this reveals helpful for someone.