Preventing multiprocessing "runaway bugs"

Question

The OS will not like it if you use multiprocessing and accidentally end up creating processes without limit.

Is there any simple solution that prevents this from happening (say, by limiting total number of processes, either in Python or in the OS)?

I use Windows, and it behaves really badly (requires hard reboot) when I make a mistake like that. So I'd love it if there's some code that I can wrap around / add to my application and prevent this from happening.

Answer 1

What you can do is create a short 'trip-wire' type module and import it as well as multiprocessing. The trip-wire module will raise an exception if it detects a multiprocessing infinite loop.

Mine looks like this:

#mp_guard.py
"""tracks invocation by creating an environment variable; if that
variable exists when next called a loop is in progress"""

import os

class Brick(Exception):
    def __init__(self):
        Exception.__init__(self, "Your machine just narrowly avoided becoming"
                                 " a brick!")

if 'MP_GUARD' in os.environ:
    raise Brick

os.environ['MP_GUARD'] = 'active'

And in the main .py file:

import mp_guard