Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Why is this message printed more than once during multiprocessing with concurrent.futures.ProcessPoolExecuter()?

The statement "I should appear only once" should appear only once. I am not able to understand why it appears 3 more times... It's clear to me that my code is executing 3 further processes. But in these 3 processes only funktion0() is getting called. Why does the statement "I should appear only once" get included in these extra 3 processes? Could someone explain?

Code:

from datetime import datetime
#print(datetime.now().time())

from time import time, sleep
#print(time())
print("I should appear only once")
from concurrent import futures


def funktion0(arg0):
    sleep(arg0)
    print(f"ich habe {arg0} sek. gewartet, aktuelle Zeit: {datetime.now().time()}")

if __name__=="__main__":

    with futures.ProcessPoolExecutor(max_workers=3) as obj0:
        obj0.submit(funktion0, 5)
        obj0.submit(funktion0, 10)
        obj0.submit(funktion0, 15)
        obj0.submit(funktion0, 20)
        print("alle Aufgaben gestartet")

    print("alle Aufgaben erledigt")

Expected output:

I should appear only once
alle Aufgaben gestartet
ich habe 5 sek. gewartet, aktuelle Zeit: 18:32:51.926288
ich habe 10 sek. gewartet, aktuelle Zeit: 18:32:56.923648
ich habe 15 sek. gewartet, aktuelle Zeit: 18:33:01.921168
ich habe 20 sek. gewartet, aktuelle Zeit: 18:33:11.929370
alle Aufgaben erledigt

Actual output:

I should appear only once
alle Aufgaben gestartet
I should appear only once
I should appear only once
I should appear only once
ich habe 5 sek. gewartet, aktuelle Zeit: 18:32:51.926288
ich habe 10 sek. gewartet, aktuelle Zeit: 18:32:56.923648
ich habe 15 sek. gewartet, aktuelle Zeit: 18:33:01.921168
ich habe 20 sek. gewartet, aktuelle Zeit: 18:33:11.929370
alle Aufgaben erledigt
like image 674
vostro.beta Avatar asked Feb 02 '19 17:02

vostro.beta


People also ask

Is it possible to print during multiprocessing tasks?

print function unable while multiprocessing.Process is being run Not sure if this really is a bug, but the multiprocessing.Process (or Pool) does not allow to print during multiprocessing tasks. I've copied the example from The Python V3.2.2 documentation, library reference, multiprocessing (3rd example).

Should you use concurrent futures or multi-processing?

One point to consider is that concurrent.futures provides a couple different implementations that allow you to easily change how your computations are happening in parallel. In the next lesson, you’ll see which situations might be better suited to using either concurrent.futures or multiprocessing.

What is Multiprocessing?

Multiprocessing is the ability of the system to handle multiple processes simultaneously and independently. In a multiprocessing system, the applications are broken into smaller routines and the OS gives threads to these processes for better performance.

What is multiprocessing in Python and why is it important?

That’s why multiprocessing in Python becomes essential. The smaller task threads act like different employees, making it easier to handle and manage various processes. A multiprocessing system can be represented as: A multi-core processor, i.e., a single computing unit with multiple independent core processing units


1 Answers

It's the classic issue with windows (RuntimeError on windows trying python multiprocessing) only less dramatic.

When you're using multiprocessing on windows, the process isn't forked but duplicated (because fork doesn't exist in windows OS) using some tricky mechanism to "emulate" fork but not accurately because OS doesn't allow it (What's the best way to duplicate fork() in windows?).

So the statement is printed as many times as there are processes, unless you protect it with __name__ == "__main__"

(you can probably speed up workers startup by moving most import statements in that scope too)

like image 107
Jean-François Fabre Avatar answered Oct 13 '22 00:10

Jean-François Fabre