Python multiprocessing: object passed by value?

Question

I have been trying the following:

from multiprocessing import Pool

def f(some_list):
    some_list.append(4)
    print 'Child process: new list = ' + str(some_list)
    return True

if __name__ == '__main__':

    my_list = [1, 2, 3]
    pool = Pool(processes=4)
    result = pool.apply_async(f, [my_list])
    result.get()

    print 'Parent process: new list = ' + str(my_list)

What I get is:

Child process: new list = [1, 2, 3, 4]
Parent process: new list = [1, 2, 3]

So, it means that the my_list was passed by value since it did not mutate. So, is the rule that it is really passed by value when passed to another process? Thanks.

Answer 1

As André Laszlo said, the multiprocessing library needs to pickle all objects passed to multiprocessing.Pool methods in order to pass them to worker processes. The pickling process results in a distinct object being created in the worker process, so that changes made to the object in the worker process have no effect on the object in the parent. On Linux, objects sometimes get passed to the child process via fork inheritence (e.g. multiprocessing.Process(target=func, args=(my_list,))), but in that case you end up with a copy-on-write version of the object in the child process, so you still end up with distinct copies when you try to modify it in either process.

If you do want to share an object between processes, you can use a multiprocessing.Manager for that:

from multiprocessing import Pool, Manager

def f(some_list):
    some_list.append(4)
    print 'Child process: new list = ' + str(some_list)
    return True

if __name__ == '__main__':

    my_list = [1, 2, 3]
    m = Manager()
    my_shared_list = m.list(my_list)
    pool = Pool(processes=4)
    result = pool.apply_async(f, [my_shared_list])
    result.get()

    print 'Parent process: new list = ' + str(my_shared_list)

Output:

Child process: new list = [1, 2, 3, 4]
Parent process: new list = [1, 2, 3, 4]

Answer 2

The multiprocessing library serializes objects using pickle to pass them between processes.

This ensures safe inter-process communication, and two processes can use the "same" objects without using shared memory.