multiprocessing Pool hangs when there is a exception in any of the thread

Question

I am new to Python and trying a multiprocessing.pool program to process files, it works fine as long as there are no exceptions. If any of the thread/process gets an exception the whole program waits for the thread

snippet of the code:

cp = ConfigParser.ConfigParser()
cp.read(gdbini)
for table in cp.sections():
    jobs.append(table)
#print jobs
poolreturn = pool.map(worker, jobs)
pool.close()
pool.join()

Failure Message:

---

Traceback (most recent call last):
  File "/opt/cnet-python/default-2.6/lib/python2.6/threading.py", line 525, in __bootstrap_inner
    self.run()
  File "/opt/cnet-python/default-2.6/lib/python2.6/threading.py", line 477, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/opt/cnet-python/default-2.6/lib/python2.6/multiprocessing/pool.py", line 259, in _handle_results
    task = get()
TypeError: ('__init__() takes exactly 3 arguments (2 given)', <class 'ConfigParser.NoOptionError'>, ("No option 'inputfilename' in section: 'section-1'",))

---

I went ahead added a exception handler to terminate the process

try:
    ifile=cp.get(table,'inputfilename')
except ConfigParser.NoSectionError,ConfigParser.NoOptionError:
    usage("One of Parameter not found for"+ table)
    terminate()

but still it waits, not sure whats missing.

Answer 1

In Python 3.2+ this works as expected. For Python 2, this bug was fixed in r74545 and will be available in Python 2.7.3. In the mean time, you can use the configparser library which is a backport of the configparser from 3.2+. Check it out.