Python Multiprocessing RuntimeError on Windows

Question

I have a class function (let's call it "alpha.py") that uses multiprocessing (processes=2) to fork a process and is part of a Python package that I wrote. In a separate Python script (let's call it "beta.py"), I instantiated an object from this class and called the corresponding function that uses multiprocessing. Finally, all of this is wrapped inside a wrapper Python script (let's call this "gamma.py") that handles many different class objects and functions.

Essentially:

1. Run ./gamma.py from the command line
2. gamma.py uses subprocess and executes beta.py
3. beta.py instantiates an object from the alpha.py class and calls the function which uses multiprocessing (processes=2)

This has no problems being run on a Mac or Linux. However, it becomes a problem on a Windows machine and the error (and documentation) suggests that I should write this somewhere:

if __name__ == '__main__':
    freeze_support()

This other post also mentions doing the same thing.

However, I don't know exactly where these two lines should reside. Currently, neither alpha.py, beta.py, or gamma.py contains an if __name__ == '__main__': section. It would be great if somebody can tell me where these two lines should go and also the rationale behind it.

Answer 1

Actually, freeze_support() is not needed here. You get a RuntimeError because you create and start your new processes at the top level of your beta module.

When a new process is created using multiprocessing on Windows, a new Python interpreter will be started in this process and it will try to import the module with the target function that should be executed. This is your beta module. Now, when you import it, all your top level statements should be executed which will cause a new process to be created and started again. And then, recursively, another process from that process, and so on and so forth.

This is most likely not what you want, thus new processes should be initialized and started only once, when you run beta.py directly with a subprocess.

if __name__ == '__main__': should be placed in beta.py, then move initialization and start code for your new processes in this section. After that, when beta.py will be imported and not run directly, no new process will be started and you will not see any side effects.