Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

python multiprocessing on windows, if __name__ == "__main__"

Tags:

Running python 2.7 on windows 7 (64bit).

When reading the docs for library module multiprocessing, it states several times the importance of the __main__ module, including the conditional (especially in Windows):

if __name__ == "__main__":     # create Process() here 

My understanding, is that you don't want to create Process() instances in the global namespace of the module (because when the child process imports the module, he will spawn yet another inadvertently).

I do not have to place Process managers at the very top level of my package execution hierarchy though (execution in the PARENT). As long as my Process()'s are created, managed, and terminated in a class method, or even in a function closure. Just not in the toplevel module namespace.

Am I understanding this warning/requirement correctly?


EDIT

After the first two responses, I add this quotation. This is in the introduction for Section 16.6 multiprocessing from the 2.7 docs.

Note: Functionality within this package requires that the __main__ module be importable by the children. This is covered in Programming guidelines however it is worth pointing out here.This means that some examples, such as the multiprocessing.Pool examples will not work in the interactive interpreter...

like image 266
user2097818 Avatar asked Nov 26 '13 16:11

user2097818


2 Answers

You do not have to call Process() from the "top level" of the module. It is perfectly fine to call Process from a class method.

The only caveat is that you can not allow Process() to be called if or when the module is imported.

Since Windows has no fork, the multiprocessing module starts a new Python process and imports the calling module. If Process() gets called upon import, then this sets off an infinite succession of new processes (or until your machine runs out of resources). This is the reason for hiding calls to Process() inside

if __name__ == "__main__" 

since statements inside this if-statement will not get called upon import.

like image 56
unutbu Avatar answered Oct 13 '22 05:10

unutbu


__name__ is only ever equal to "__main__" if the script has been executed directly, either via python foo.py or python -m foo. This ensures that Process() will not be called if the script is imported as a module instead.

like image 42
Ignacio Vazquez-Abrams Avatar answered Oct 13 '22 04:10

Ignacio Vazquez-Abrams