Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Python multiprocessing and an imported module

I have two processing running that access an imported module like that:

import foo

def bar():
  while True:
    foo.a = True

def baz():
  while True:
    print foo.a

p1 = Process(target=bar)
p2 = Process(target=baz)
p1.start()
p2.start()

It seems that each process have their own instance of module foo, bar() changes value to True, but in baz() it's False. Any workaround?

like image 644
akalikin Avatar asked Aug 19 '15 13:08

akalikin


People also ask

How do I import a multiprocessing module in Python?

In this example, at first we import the Process class then initiate Process object with the display() function. Then process is started with start() method and then complete the process with the join() method. We can also pass arguments to the function using args keyword.

How many processes should be running Python multiprocessing?

If we are using the context manager to create the process pool so that it is automatically shutdown, then you can configure the number of processes in the same manner. The number of workers must be less than or equal to 61 if Windows is your operating system.

Does Python multiprocessing use shared memory?

Python 3.8 introduced a new module multiprocessing. shared_memory that provides shared memory for direct access across processes. My test shows that it significantly reduces the memory usage, which also speeds up the program by reducing the costs of copying and moving things around.


1 Answers

Unlike threads, separate processes do not share memory. There are ways, however, to share data between separate processes. One way is to use a mp.Value:

foo.py:

import multiprocessing as mp
a = mp.Value('b', False)

then the script

import time
import foo
import multiprocessing as mp

def bar():
    foo.a.value = True

def baz():
    for i in range(10**5):
        print foo.a.value

if __name__ == '__main__':
    p2 = mp.Process(target=baz)
    p2.start()
    time.sleep(0.5)
    p1 = mp.Process(target=bar)
    p1.start()

yields

0
0
0
...
1
1
1
...

For sharing a boolean value, an mp.Event is perhaps a better option:

foo.py:

import multiprocessing as mp
a = mp.Event()

script.py:

def bar():
    foo.a.set()

def baz():
    for i in range(10**5):
        print foo.a.is_set()

yields

False
False
False
...
True
True
True
... 
like image 121
unutbu Avatar answered Sep 19 '22 12:09

unutbu