Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

multiprocessing breaks in interactive mode

I have the following code

from multiprocessing import Process, Queue
from queue import Empty
from time import sleep

def f(q):
    n = 100000000
    while n != 100000000 // 2:
        n -= 1
    q.put("the awkening!")
    print("my work here is done")

def main():
    q = Queue()
    p = Process(target=f, args=(q,))
    p.start()
    while True:
        try:
            print(q.get(block=False))
            raise systemexit
        except Empty:
            print("i found nothing :(")
            sleep(2)
    p.join()

If I add

if __name__ == '__main__':
     main()

To the end then use python script_name.py to run it, everything works fine. However, if I just run the scirpt using python -i script_name.py then run main() Python complains:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Python34\lib\multiprocessing\spawn.py", line 98, in spawn_main
    exitcode = _main(fd)
  File "C:\Python34\lib\multiprocessing\spawn.py", line 108, in _main
    self = pickle.load(from_parent)
AttributeError: Can't get attribute 'f' on <module '__main__' (built-in)>

The error comes from the child process, the main process runs fine.

This is not a big deal, but I wonder why this happens, also it would be nice if it works in interactive mode

like image 221
XrXr Avatar asked Jun 28 '14 12:06

XrXr


People also ask

What is Chunksize in multiprocessing?

The chunksize argument is the same as the one used by the map() method. For very long iterables using a large value for chunksize can make the job complete much faster than using the default value of 1. — multiprocessing — Process-based parallelism. In the case of Pool.

What is multiprocessing Freeze_support?

multiprocessing. freeze_support() This function will allow a frozen program to create and start new processes via the multiprocessing. Process class when the program is frozen for distribution on Windows. If the function is called and the program is not frozen for distribution, then it has no effect.

How many processes should be running Python multiprocessing?

If we are using the context manager to create the process pool so that it is automatically shutdown, then you can configure the number of processes in the same manner. The number of workers must be less than or equal to 61 if Windows is your operating system.


1 Answers

The multiprocessing documentation discusses this:

Note

Functionality within this package requires that the __main__ module be importable by the children. This is covered in Programming guidelines however it is worth pointing out here. This means that some examples, such as the multiprocessing.Pool examples will not work in the interactive interpreter.

My understanding is that __main__ is defined very differently in the context of an interactive session (as it is associated with the shell, not the file that is running).

like image 174
sapi Avatar answered Sep 28 '22 02:09

sapi