Appending to the same list from different processes using multiprocessing

Question

I need to append objects to one list L from different processes using multiprocessing , but it returns empty list. How can I let many processes append to list L using multiprocessing?

#!/usr/bin/python from multiprocessing import Process  L=[] def dothing(i,j):     L.append("anything")     print i  if __name__ == "__main__":     processes=[]     for i in range(5):         p=Process(target=dothing,args=(i,None))         p.start()         processes.append(p)     for p in processes:         p.join()  print L 

Answer 1

Global variables are not shared between processes.

You need to use multiprocessing.Manager.list:

from multiprocessing import Process, Manager  def dothing(L, i):  # the managed list `L` passed explicitly.     L.append("anything")  if __name__ == "__main__":     with Manager() as manager:         L = manager.list()  # <-- can be shared between processes.         processes = []         for i in range(5):             p = Process(target=dothing, args=(L,i))  # Passing the list             p.start()             processes.append(p)         for p in processes:             p.join()         print L 

See Sharing state between processes¶ (Server process part).

Answer 2

Falsetru's answer worked.

But still, the list was not accessible beyond the with Manager() as manager: two changes were needed:

1. adding L = [] in front of the if __name__ == "__main__": statement. Must be added as for some reason the last print(L) (the one outside of if) is executed Processes + 1 times. This returns an error that L is not defined and the code breaks.

2. adding L = list(L)after the p.join() statement. This step is needed to change Manager.list to regular Python.list - otherwise calls to the manager.list return errors that object not readable.

from multiprocessing import Process, Manager  def dothing(L, i):  # the managed list `L` passed explicitly.     for j in range(5):         text = "Process " + str(i) + ", Element " + str(j)         L.append(text)  L = []   if __name__ == "__main__":     with Manager() as manager:         L = manager.list()  # <-- can be shared between processes.         processes = []          for i in range(5):             p = Process(target=dothing, args=(L,i,))  # Passing the list             p.start()             processes.append(p)          for p in processes:             p.join()          L = list(L)          print("Within WITH")         print(L)      print("Within IF")     print(L)  print("Outside of IF") print(L) 

Output:

Outside of IF [] Outside of IF [] Outside of IF [] Outside of IF [] Outside of IF [] Outside of IF [] Within WITH ['Process 2, Element 0','Process 2, Element 1', 'Process 2, Element 2', 'Process 2, Element 3', 'Process 2, Element 4', 'Process 1, Element 0',  'Process 1, Element 1', 'Process 1, Element 2', 'Process 1, Element 3',  'Process 1, Element 4', 'Process 0, Element 0', 'Process 0, Element 1',  'Process 0, Element 2', 'Process 0, Element 3', 'Process 0, Element 4',  'Process 4, Element 0', 'Process 4, Element 1', 'Process 4, Element 2',  'Process 4, Element 3', 'Process 4, Element 4', 'Process 3, Element 0',  'Process 3, Element 1', 'Process 3, Element 2', 'Process 3, Element 3',  'Process 3, Element 4']  Within IF ['Process 2, Element 0','Process 2, Element 1', 'Process 2, Element 2',  'Process 2, Element 3', 'Process 2, Element 4', 'Process 1, Element 0',  'Process 1, Element 1', 'Process 1, Element 2', 'Process 1, Element 3',  'Process 1, Element 4', 'Process 0, Element 0', 'Process 0, Element 1',  'Process 0, Element 2', 'Process 0, Element 3', 'Process 0, Element 4',  'Process 4, Element 0', 'Process 4, Element 1', 'Process 4, Element 2', 'Process 4, Element 3', 'Process 4, Element 4', 'Process 3, Element 0',  'Process 3, Element 1', 'Process 3, Element 2', 'Process 3, Element 3',  'Process 3, Element 4']  Outside of IF ['Process 2, Element 0','Process 2, Element 1', 'Process 2, Element 2',  'Process 2, Element 3', 'Process 2, Element 4', 'Process 1, Element 0',  'Process 1, Element 1', 'Process 1, Element 2', 'Process 1, Element 3',  'Process 1, Element 4', 'Process 0, Element 0', 'Process 0, Element 1',  'Process 0, Element 2', 'Process 0, Element 3', 'Process 0, Element 4',  'Process 4, Element 0', 'Process 4, Element 1', 'Process 4, Element 2',  'Process 4, Element 3', 'Process 4, Element 4', 'Process 3, Element 0',  'Process 3, Element 1', 'Process 3, Element 2', 'Process 3, Element 3',  'Process 3, Element 4']