Cannot append items to multiprocessing shared list

Question

I'm using multiprocessing to create sub-process to my application. I also share a dictionary between the process and the sub-process.

Example of my code:

Main process:

from multiprocessing import Process, Manager
manager = Manager()
shared_dict = manager.dict()
p = Process(target=mysubprocess, args=(shared_dict,))
p.start()
p.join()
print shared_dict

my sub-process:

def mysubprocess(shared_dict):
  shared_dict['list_item'] = list()
  shared_dict['list_item'].append('test')
  print shared_dict

In both cases the printed value is : {'list_item': []}

What could be the problem? Thanks

Answer 1

Manager.dict will give you a dict where direct changes will be propagated between the processes, but it doesn't detect if you change objects contained in the dict (like the list stored under "list_item"). See the note at the bottom of the SyncManager documentation:

Note: Modifications to mutable values or items in dict and list proxies will not be propagated through the manager, because the proxy has no way of knowing when its values or items are modified. To modify such an item, you can re-assign the modified object to the container proxy.

So in your example the list gets synced when you set it in the dict, but the append doesn't trigger another sync.

You can get around that by re-assigning the key in the dict:

from multiprocessing import Process, Manager

def mysubprocess(shared_dict):
    item = shared_dict['list_item'] = list()
    item.append('test')
    shared_dict['list_item'] = item
    print 'subprocess:', shared_dict

manager = Manager()
shared_dict = manager.dict()
p = Process(target=mysubprocess, args=(shared_dict,))
p.start()
p.join()
print 'main process:', shared_dict

But that might get inefficient if the list is going to grow long - the whole list will be serialised and sent to the manager process for each append. A better way in that case would be to make the shared list directly with SyncManager.list (although you'll still have the same problem if the elements of the list are mutable - you need to reset them in the list to send them between the processes).