Python - Multiprocessing.processes become copies of the main process when run from executable

Question

I just discovered a bizarre bug in my program related to its use of Python's multiprocessing module. Everything works fine when I run the program from the source on my machine. But I've been building it into an executable using pyinstaller, and for some reason the behavior of multiprocessing changes drastically when I run the executable built from my code. Specifically, when I try to run the multiprocessing part of my code, rather than do what it's supposed to, what appears to be a copy of my program's main window pops up, one for each process. Even worse, they reopen if they are closed manually, presumably because they are part of a multiprocessing.pool. No error messages are printed, and once created all the windows just sit there doing nothing. What could be happening to cause this?

Answer 1

On Windows, multiprocessing tries to emulate the Unix fork() system call by starting new instances of your executable, and execute its child process routine (multiprocessing.forking.main()) therein. With the standard Python interpreter (python.exe), multiprocessing can pass the -c parameter to run custom code. For custom executables, however, this is not be possible since the executable will most probably not support the same command line options as python.exe.

The freeze_support() function sidesteps this problem by executing the child process routine explicitely, and terminate the interpreter by calling sys.exit(). If you forget to call freeze_support(), the new process does not know that it is a child process and runs the main application logic. In your case, this will pop up another main GUI window.

Since starting yet another child process from the newly created process will cause infinite recursion, multiprocessing tries to prevent this by checking the sys.frozen attribute and raise a RuntimeError if freeze_support() was not called. In your case, it seems that user interaction is required to spawn the processes, therefore there is no infinite recursion and no RuntimeError.

By convention, sys.frozen is only set for automatically generated executables as created by py2exe or PyInstaller. It is important to understand this logic and set sys.frozen to True when one wants to embed Python in a custom executable that should support multiprocessing under windows.