Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

RuntimeError on windows trying python multiprocessing

People also ask

Does Python multiprocessing work on Windows?

The multiprocessing package offers both local and remote concurrency, effectively side-stepping the Global Interpreter Lock by using subprocesses instead of threads. Due to this, the multiprocessing module allows the programmer to fully leverage multiple processors on a given machine. It runs on both Unix and Windows.

Is not going to be frozen to produce an executable RuntimeError?

RuntimeError When Starting a Child Process is not going to be frozen to produce an executable. This will happen on Windows and MacOS where the default start method is 'spawn'. It may also happen when you configure your program to use the 'spawn' start method on other platforms.

How lock in multiprocessing Python?

Python provides a mutual exclusion lock for use with processes via the multiprocessing. Lock class. An instance of the lock can be created and then acquired by processes before accessing a critical section, and released after the critical section. Only one process can have the lock at any time.

Does multiprocessing work in Python?

Multiprocess is a Python package that supports spawning processing tasks using an API similar to the Python threading module. In addition, the multiprocessing package supports concurrency in both local and remote types, allowing you to bypass the global interpreter lock that comes with threading.


On Windows the subprocesses will import (i.e. execute) the main module at start. You need to insert an if __name__ == '__main__': guard in the main module to avoid creating subprocesses recursively.

Modified testMain.py:

import parallelTestModule

if __name__ == '__main__':    
    extractor = parallelTestModule.ParallelExtractor()
    extractor.runInParallel(numProcesses=2, numThreads=4)

Try putting your code inside a main function in testMain.py

import parallelTestModule

if __name__ ==  '__main__':
  extractor = parallelTestModule.ParallelExtractor()
  extractor.runInParallel(numProcesses=2, numThreads=4)

See the docs:

"For an explanation of why (on Windows) the if __name__ == '__main__' 
part is necessary, see Programming guidelines."

which say

"Make sure that the main module can be safely imported by a new Python interpreter without causing unintended side effects (such a starting a new process)."

... by using if __name__ == '__main__'


Though the earlier answers are correct, there's a small complication it would help to remark on.

In case your main module imports another module in which global variables or class member variables are defined and initialized to (or using) some new objects, you may have to condition that import in the same way:

if __name__ ==  '__main__':
  import my_module

As @Ofer said, when you are using another libraries or modules, you should import all of them inside the if __name__ == '__main__':

So, in my case, ended like this:

if __name__ == '__main__':       
    import librosa
    import os
    import pandas as pd
    run_my_program()

hello here is my structure for multi process

from multiprocessing import Process
import time


start = time.perf_counter()


def do_something(time_for_sleep):
    print(f'Sleeping {time_for_sleep} second...')
    time.sleep(time_for_sleep)
    print('Done Sleeping...')



p1 = Process(target=do_something, args=[1])
p2 = Process(target=do_something, args=[2])


if __name__ == '__main__':
    p1.start()
    p2.start()

    p1.join()
    p2.join()

    finish = time.perf_counter()
    print(f'Finished in {round(finish-start,2 )} second(s)')

you don't have to put imports in the if __name__ == '__main__':, just running the program you wish to running inside