Python Multiprocessing Sharing Variables Between Processes

Question

I'm trying to write a multiprocessing program which shares one or more variables (values or matrix) between the child processes. In my current test program I'm trying to spawn two processes, each sharing the num variable. Each adds 1 to the variable and then prints. However, whenever I try to run the program it tells me a TypeError has occurred, saying 'Synchronized' object is not iterable. How can I get this to work?

The code is shown below:

import multiprocessing
import os
import time

def info(title):
    print(title)
    print('module name:', __name__)
    print('parent process:', os.getppid())
    print('process id:', os.getpid())

def f(num):
    while True:
        time.sleep(1.5)
        num = num + 1
        print("process 1: %i \n" % num)
        if num > 50:
            break

def f2(num):
    while True:
        num= num + 1
        print("process 2: %i \n" % num)
        time.sleep(1.9)
        if num > 50:
            break

if __name__ == '__main__':
    data = multiprocessing.Value('i',0)
    p = multiprocessing.Process(target=f, args=(data))
    j = multiprocessing.Process(target=f2, args=(data))
    p.start()
    j.start()

"Synchronized' object is not iterable results from running the program when it tries to create the first process: p = multiprocessing.Process(target=f, args=(data))

I'm not sure whether using a queue would work as I'd eventually like to have a program which has a looping process and another which occasionally grabs the most recent result returned from the looping process.

Answer 1

You have several issues with your code:

1. When you create a multiprocessing.Value instance (num in your case), you must use the value attribute of that instance to read or write actual value of that shared variable.
2. Incrementing such an instance, even if you replace num.value = num.value + 1 with num.value += 1 as I have done, is not an atomic operation. It is equivalent to temp = num.value; temp += 1; num.value = temp, which means that this incrementing must take place under control of the lock provided for such synchronized instances to ensure that the value is truly incremeneted. Otherwise, two processes may read the same value and increment it to the same final value and you will have only incremented the value once instead of twice.
3. The args parameter to the multiprocessing.Process initializer should be an iterable where each element of the iterable becomes an argument to your worker functions f and f2. When you specify args=(data), the parentheses has no effect and is equivalent to simply specifying args=data and data is not an iterable. You needed to have args=(data,). Note the comma (',') such that (data,) is now a tuple (i.e. an iterable) containing the single element data instead of a parenthesized expression.

import multiprocessing
import os
import time

def info(title):
    print(title)
    print('module name:', __name__)
    print('parent process:', os.getppid())
    print('process id:', os.getpid())

def f(num):
    while True:
        time.sleep(1.5)
        with num.get_lock():
            num.value += 1
            print("process 1: %i \n" % num.value)
            if num.value > 50:
                break

def f2(num):
    while True:
        with num.get_lock():
            num.value += 1
            print("process 2: %i \n" % num.value)
            time.sleep(1.9)
            if num.value > 50:
                break

if __name__ == '__main__':
    data = multiprocessing.Value('i',0)
    p = multiprocessing.Process(target=f, args=(data,))
    j = multiprocessing.Process(target=f2, args=(data,))
    p.start()
    j.start()
    p.join()
    j.join()
    print(data.value)

Prints:

...
process 2: 47

process 1: 48

process 2: 49

process 1: 50

process 2: 51

process 1: 52

52

But note that in the above code, each process acquires and hold the lock for a very long time shutting out any other process from acquiring the lock. We can minimize how long the lock is held in the following way (and so the program will complete sooner):

import multiprocessing
import os
import time

def info(title):
    print(title)
    print('module name:', __name__)
    print('parent process:', os.getppid())
    print('process id:', os.getpid())

def f(num):
    while True:
        time.sleep(1.5)
        with num.get_lock():
            num.value += 1
            saved_value = num.value
        print("process 1: %i \n" % saved_value)
        if saved_value > 50:
            break

def f2(num):
    while True:
        with num.get_lock():
            num.value += 1
            saved_value = num.value
        print("process 2: %i \n" % saved_value)
        time.sleep(1.9)
        if saved_value > 50:
            break

if __name__ == '__main__':
    data = multiprocessing.Value('i',0)
    p = multiprocessing.Process(target=f, args=(data,))
    j = multiprocessing.Process(target=f2, args=(data,))
    p.start()
    j.start()
    p.join()
    j.join()