Python3 filling a dictionary concurrently

Question

I want to fill a dictionary in a loop. Iterations in the loop are independent from each other. I want to perform this on a cluster with thousands of processors. Here is a simplified version of what I tried and need to do.

import multiprocessing

class Worker(multiprocessing.Process):
   def setName(self,name):
       self.name=name
   def run(self):
       print ('In %s' % self.name)
       return

if __name__ == '__main__':
   jobs = []
   names=dict()
   for i in range(10000):
       p = Worker()
       p.setName(str(i))
       names[str(i)]=i
       jobs.append(p)
       p.start()
   for j in jobs:
       j.join()

I tried this one in python3 on my own computer and received the following error:

    ..
    In 249
    Traceback (most recent call last):
      File "test.py", line 16, in <module>
        p.start()
      File         "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/multiprocessing/process.py", line 105, in start
    In 250
        self._popen = self._Popen(self)
      File         "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/multiprocessing/context.py", line 212, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
      File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/multiprocessing/context.py", line 267, in _Popen
return Popen(process_obj)
      File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/multiprocessing/popen_fork.py", line 20, in __init__
self._launch(process_obj)
      File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/multiprocessing/popen_fork.py", line 66, in _launch
parent_r, child_w = os.pipe()
    OSError: [Errno 24] Too many open files

Is there any better way to do this?

Answer 1

multiprocessing talks to its subprocesses via pipes. Each subprocesses requires two open file descriptors, one for read and one for write. If you launch 10000 workers, you'll end opening 20000 file descriptors which exceeds the default limit on OS X (which your paths indicate you're using).

You can fix the issue by raising the limit. See https://superuser.com/questions/433746/is-there-a-fix-for-the-too-many-open-files-in-system-error-on-os-x-10-7-1 for details - basically, it amounts to setting two sysctl knobs and upping your shell's ulimit setting.

Answer 2

You are spawning 10000 processes at once at the moment. That really isn't a good idea.
The error you see is most definitely raised because the multiprocessing module (seem to) use pipes for the Inter Proccess Communication and there is a limit of open pipes/FDs.

I suggest using an python interpreter without a Global interpreter lock like Jython or IronPython and just replace the multiprocessing module with the threading one.

---

If you still want to use the multiprocessing module, you could use an Proccess Pool like this to collect the return values:

from multiprocessing import Pool

def worker(params):
    name, someArg = params
    print ('In %s' % name)
    # do something with someArg here
    return (name, someArg)

if __name__ == '__main__':
    jobs = []
    names=dict()
    # Spawn 100 worker processes 
    pool = Pool(processes=100)
    # Fill with real data
    task_dict = dict(('name_{}'.format(i), i) for i in range(1000))
    # Process every task via our pool
    results = pool.map(worker, task_dict.items())
    # And convert the rsult to a dict
    results = dict(results)
    print (results)

This should work with minimal changes for the threading module, too.