How to launch crashing (rarely) application in subprocess

Question

I'm having python application which needs to execute proprietary application (which crashes from time to time) about 20 000 times a day.

The problem is when application crashes, Windows automatically triggers WerFault which will keep program hanging, thus python's subprocess.call() will wait forever for user input (that application has to run on weekends, on holidays, 24/7... so this is not acceptable).

If though about using sleep; poll; kill; terminate but that would mean losing ability to use communicate(), application can run from few miliseconds to 2 hours, so setting fixed timeout will be ineffective

I also tried turning on automatic debugging (use a script which would take a crash dump of an application and terminate id), but somehow this howto doesn't work on my server (WerFault still appears and waits for user input).

Several other tutorials like this didn't take any effect either.

Question: is there a way how to prevent WerFault from displaying (waiting for user input)? _{this is more system then programming question}

Alternative question: is there a graceful way in python how to detect application crash (whether WerFault was displayed)

Answer 1

Simple (and ugly) answer, monitor for WerFault.exe instances from time to time, specially the one associated with the PID of the offending application. And kill it. Dealing with WerFault.exe is complicated but you don't want to disable it -- see Windows Error Reporting service.

1. Get a list of processes by name that match WerFault.exe. I use psutil package. Be careful with psutil because processes are cached, use psutil.get_pid_list().
2. Decode its command line by using argparse. This might be overkill but it leverages existing python libraries.
3. Identify the process that is holding your application according to its PID.

This is a simple implementation.

def kill_proc_kidnapper(self, child_pid, kidnapper_name='WerFault.exe'):
    """
    Look among all instances of 'WerFault.exe' process for an specific one
    that took control of another faulting process.
    When 'WerFault.exe' is launched it is specified the PID using -p argument:

    'C:\\Windows\\SysWOW64\\WerFault.exe -u -p 5012 -s 68'
                             |               |
                             +-> kidnapper   +-> child_pid

    Function uses `argparse` to properly decode process command line and get
    PID. If PID matches `child_pid` then we have found the correct parent
    process and can kill it.
    """
    parser = argparse.ArgumentParser()
    parser.add_argument('-u', action='store_false', help='User name')
    parser.add_argument('-p', type=int, help='Process ID')
    parser.add_argument('-s', help='??')

    kidnapper_p = None
    child_p = None

    for proc in psutil.get_pid_list():
        if kidnapper_name in proc.name:
            args, unknown_args = parser.parse_known_args(proc.cmdline)
            print proc.name, proc.cmdline

            if args.p == child_pid:
                # We found the kidnapper, aim.
                print 'kidnapper found: {0}'.format(proc.pid)
                kidnapper_p = proc

    if psutil.pid_exists(child_pid):
        child_p = psutil.Process(child_pid)

    if kidnapper_p and child_pid:
        print 'Killing "{0}" ({1}) that kidnapped "{2}" ({3})'.format(
            kidnapper_p.name, kidnapper_p.pid, child_p.name, child_p.pid)
        self.taskkill(kidnapper_p.pid)
        return 1
    else:
        if not kidnapper_p:
            print 'Kidnapper process "{0}" not found'.format(kidnapper_name)
        if not child_p:
            print 'Child process "({0})" not found'.format(child_pid)

    return 0

Now, taskkill function invokes taskkill commmand with correct PID.

def taskkill(self, pid):
    """
    Kill task and entire process tree for this process
    """
    print('Task kill for PID {0}'.format(pid))
    cmd = 'taskkill /f /t /pid {0}'.format(pid)
    subprocess.call(cmd.split())