Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to handle initializer error in multiprocessing.Pool?

When initializer throw Error like below, script won't stop.
I would like to abort before starting main process(do not run 'do_something').

from multiprocessing import Pool
import contextlib

def initializer():
    raise Exception("init failed")

def do_something(args):
    # main process
    pass

pool = Pool(1, initializer=initializer)
with contextlib.closing(pool):
    try:
        pool.map_async(do_something, [1]).get(100)
    except:
        pool.terminate()

The never stopping stacktrace on console is below

...
Exception: init failed
Process ForkPoolWorker-18:
Traceback (most recent call last):
  File "/home/hoge/anaconda3/lib/python3.6/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/home/hoge/anaconda3/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/hoge/anaconda3/lib/python3.6/multiprocessing/pool.py", line 103, in worker
    initializer(*initargs)
  File "hoge.py", line 5, in initializer
    raise Exception("init failed")
Exception: init failed
...

My workaround is suppressing initializer error and return at the beginning of the main process by using global flag like below.
But I would like to learn better one.

def initializer():
    try:
        raise Exception("init failed")
    except:
        global failed
        failed = True

def do_something(args):
    global failed
    if failed:
        # skip when initializer failed
        return
    # main process
like image 776
tabata Avatar asked Jun 23 '26 20:06

tabata


1 Answers

After navigating through the implementation of multiprocessing using PyCharm, I'm convinced that there is no better solution, because Pool started a thread to _maintain_pool() by _repopulate_pool() if any worker process exists--either accidentally or failed to initialize.

Check this out: Lib/multiprocessing/pool.py line 244 @ab74504

like image 182
TingQian LI Avatar answered Jun 25 '26 09:06

TingQian LI



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!