Menu
I need to append objects to one list L from different processes using multiprocessing , but it returns empty list. How can I let many processes append to list L using multiprocessing?
#!/usr/bin/python from multiprocessing import Process L=[] def dothing(i,j): L.append("anything") print i if __name__ == "__main__": processes=[] for i in range(5): p=Process(target=dothing,args=(i,None)) p.start() processes.append(p) for p in processes: p.join() print L 
671
Key Takeaways. Python is NOT a single-threaded language. Python processes typically use a single thread because of the GIL. Despite the GIL, libraries that perform computationally heavy tasks like numpy, scipy and pytorch utilise C-based implementations under the hood, allowing the use of multiple cores.
Synchronization between processes Multiprocessing is a package which supports spawning processes using an API. This package is used for both local and remote concurrencies. Using this module, programmer can use multiple processors on a given machine. It runs on Windows and UNIX os.
As we have seen, the Pool allocates only executing processes in memory and the process allocates all the tasks in memory, so when the task number is small, we can use process class and when the task number is large, we can use the pool.
Multiprocessing is the ability of a system to run multiple processors at one time. If you had a computer with a single processor, it would switch between multiple processes to keep all of them running. However, most computers today have at least a multi-core processor, allowing several processes to be executed at once.
Global variables are not shared between processes.
You need to use multiprocessing.Manager.list:
from multiprocessing import Process, Manager def dothing(L, i): # the managed list `L` passed explicitly. L.append("anything") if __name__ == "__main__": with Manager() as manager: L = manager.list() # <-- can be shared between processes. processes = [] for i in range(5): p = Process(target=dothing, args=(L,i)) # Passing the list p.start() processes.append(p) for p in processes: p.join() print L See Sharing state between processes¶ (Server process part).
151
Falsetru's answer worked.
But still, the list was not accessible beyond the with Manager() as manager: two changes were needed:
adding L = [] in front of the if __name__ == "__main__": statement. Must be added as for some reason the last print(L) (the one outside of if) is executed Processes + 1 times. This returns an error that L is not defined and the code breaks.
adding L = list(L)after the p.join() statement. This step is needed to change Manager.list to regular Python.list - otherwise calls to the manager.list return errors that object not readable.
from multiprocessing import Process, Manager def dothing(L, i): # the managed list `L` passed explicitly. for j in range(5): text = "Process " + str(i) + ", Element " + str(j) L.append(text) L = [] if __name__ == "__main__": with Manager() as manager: L = manager.list() # <-- can be shared between processes. processes = [] for i in range(5): p = Process(target=dothing, args=(L,i,)) # Passing the list p.start() processes.append(p) for p in processes: p.join() L = list(L) print("Within WITH") print(L) print("Within IF") print(L) print("Outside of IF") print(L) Output:
Outside of IF [] Outside of IF [] Outside of IF [] Outside of IF [] Outside of IF [] Outside of IF [] Within WITH ['Process 2, Element 0','Process 2, Element 1', 'Process 2, Element 2', 'Process 2, Element 3', 'Process 2, Element 4', 'Process 1, Element 0', 'Process 1, Element 1', 'Process 1, Element 2', 'Process 1, Element 3', 'Process 1, Element 4', 'Process 0, Element 0', 'Process 0, Element 1', 'Process 0, Element 2', 'Process 0, Element 3', 'Process 0, Element 4', 'Process 4, Element 0', 'Process 4, Element 1', 'Process 4, Element 2', 'Process 4, Element 3', 'Process 4, Element 4', 'Process 3, Element 0', 'Process 3, Element 1', 'Process 3, Element 2', 'Process 3, Element 3', 'Process 3, Element 4'] Within IF ['Process 2, Element 0','Process 2, Element 1', 'Process 2, Element 2', 'Process 2, Element 3', 'Process 2, Element 4', 'Process 1, Element 0', 'Process 1, Element 1', 'Process 1, Element 2', 'Process 1, Element 3', 'Process 1, Element 4', 'Process 0, Element 0', 'Process 0, Element 1', 'Process 0, Element 2', 'Process 0, Element 3', 'Process 0, Element 4', 'Process 4, Element 0', 'Process 4, Element 1', 'Process 4, Element 2', 'Process 4, Element 3', 'Process 4, Element 4', 'Process 3, Element 0', 'Process 3, Element 1', 'Process 3, Element 2', 'Process 3, Element 3', 'Process 3, Element 4'] Outside of IF ['Process 2, Element 0','Process 2, Element 1', 'Process 2, Element 2', 'Process 2, Element 3', 'Process 2, Element 4', 'Process 1, Element 0', 'Process 1, Element 1', 'Process 1, Element 2', 'Process 1, Element 3', 'Process 1, Element 4', 'Process 0, Element 0', 'Process 0, Element 1', 'Process 0, Element 2', 'Process 0, Element 3', 'Process 0, Element 4', 'Process 4, Element 0', 'Process 4, Element 1', 'Process 4, Element 2', 'Process 4, Element 3', 'Process 4, Element 4', 'Process 3, Element 0', 'Process 3, Element 1', 'Process 3, Element 2', 'Process 3, Element 3', 'Process 3, Element 4'] If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
