The standard behavior of multiprocessing on Windows is to import the __main__ module into child processes when spawned.
For large projects with many imports, this can significantly slow down the child process startup, not to mention the extra resources consumed. It seems very inefficient for cases where the child process will run a self-contained task that only uses a small subset of those imports.
Is there a way to explicitly specify the imports for the child processes? If not the multiprocessing library, is there an alternative?
While I'm specifically interested in Python 3, answers for Python 2 may be useful for others.
I've confirmed that the approach suggested by Lie Ryan works, as shown by the following example:
import sys
import types
def imports():
for name, val in globals().items():
if isinstance(val, types.ModuleType):
yield val.__name__
def worker():
print('Worker modules:')
print('\n'.join(imports()))
if __name__ == '__main__':
import multiprocessing
print('Main modules:')
print('\n'.join(imports()))
print()
p = multiprocessing.Process(target=worker)
p.start()
p.join()
Output:
Main modules:
builtins
sys
types
multiprocessing
Worker modules:
sys
types
However, I don't think I can sell the rest of my team on wrapping the top-level script in if __name__ == '__main__' just to enable a small feature deep in the codebase. Still holding out hope that there's a way to do this without top-level changes.
The docs you linked tells you:
Make sure that the main module can be safely imported by a new Python interpreter without causing unintended side effects (such a starting a new process).
...Instead one should protect the “entry point” of the program by using
if __name__ == '__main__':as follows:...
You can also put import statements inside the if-block, then those import statements will only get executed when you run the __main__.py as a program but not when __main__.py is being imported.
<flame>Either that or switch to use a real OS that supports real fork()-ing</flame>
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With