Python multiprocessing, passing an object reference containig a semaphore

Question

I've a scenario like this: I've created an object of the class element containing a semaphore.

import multiprocessing as mpr

class Element(object):
    def __init__(self):
        self.sem = mpr.Semaphore()
        self.xyz = 33

def fun( ch ):
    a = ch.recv()
    print( a[0] )
    print( a[1].xyz )
    a[1].xyz = 99
    print( a[1].xyz )


el = Element()

( pa , ch ) = mpr.Pipe()
proc = mpr.Process(target=fun , args=( ch, ) )

proc.start()
pa.send( [ "Hallo" , el ])

print( el.xyz )

proc.join()

This code return this error:

  File "/usr/lib/python2.7/multiprocessing/forking.py", line 51, in assert_spawning
    ' through inheritance' % type(self).__name__
RuntimeError: Semaphore objects should only be shared between processes through inheritance

But if I remove the semaphore from the declaration of Element the code works, but the value assigned to a[1].xyz will be lost.

Now I need to synchronizes a big collection of object via semphore and multiprocessing. So there's some method for setting a semaphore in every object and passing only the reference to the main object?

import multiprocessing as mpr

class Element(object):
    def __init__(self):
        self.xyz = 33

def fun( ch ):
    a = ch.recv()
    print( a[0] )
    print( a[1].xyz )
    a[1].xyz = 99
    print( a[1].xyz )


el = Element()

( pa , ch ) = mpr.Pipe()
proc = mpr.Process(target=fun , args=( ch, ) )

proc.start()
pa.send( [ "Hallo" , el ])

print( el.xyz )

proc.join()

The second version dot't produce any error, but the value assigned to a[1].xyz = 99 will be lost in the main process.

Answer 1

I don't think you understood how the multiprocessing module works.

When you send something through the pipe, it gets pickled and then unpickled in the subprocess. This means that the subprocess actually has a copy of the original object! That's why the change is "lost". Adding a semaphore wont change anything.

If you want an object in shared memory you should use multiprocessing.Value, even though this does not handle arbitrary types. Probably multiprocessing.Manager is what you are looking for.

An other way would be to send a response to the main process providing the modified object.