Python: How to determine subprocess children have all finished running

Question

I am trying to detect when an installation program finishes executing from within a Python script. Specifically, the application is the Oracle 10gR2 Database. Currently I am using the subprocess module with Popen. Ideally, I would simply use the wait() method to wait for the installation to finish executing, however, the documented command actually spawns child processes to handle the actual installation. Here is some sample code of the failing code:

import subprocess
OUI_DATABASE_10GR2_SUBPROCESS = ['sudo',
                                 '-u',
                                 'oracle',
                                 os.path.join(DATABASE_10GR2_TMP_PATH,
                                              'database',
                                              'runInstaller'),
                                 '-ignoreSysPrereqs',
                                 '-silent',
                                 '-noconfig',
                                 '-responseFile '+ORACLE_DATABASE_10GR2_SILENT_RESPONSE]
oracle_subprocess = subprocess.Popen(OUI_DATABASE_10GR2_SUBPROCESS)
oracle_subprocess.wait()

There is a similar question here: Killing a subprocess including its children from python, but the selected answer does not address the children issue, instead it instructs the user to call directly the application to wait for. I am looking for a specific solution that will wait for all children of the subprocess. What if there are an unknown number of subprocesses? I will select the answer that addresses the issue of waiting for all children subprocesses to finish.

More clarity on failure: The child processes continue executing after the wait() command since that command only waits for the top level process (in this case it is 'sudo'). Here is a simple diagram of the known child processes in this problem: Python subprocess module -> Sudo -> runInstaller -> java -> (unknown)

Answer 1

Ok, here is a trick that will work only under Unix. It is similar to one of the answers to this question: Ensuring subprocesses are dead on exiting Python program. The idea is to create a new process group. You can then wait for all processes in the group to terminate.

pid = os.fork()
if pid == 0:
    os.setpgrp()
    oracle_subprocess = subprocess.Popen(OUI_DATABASE_10GR2_SUBPROCESS)
    oracle_subprocess.wait()
    os._exit(0)
else:
    os.waitpid(-pid)

I have not tested this. It creates an extra subprocess to be the leader of the process group, but avoiding that is (I think) quite a bit more complicated.

I found this web page to be helpful as well. http://code.activestate.com/recipes/278731-creating-a-daemon-the-python-way/

Answer 2

You can just use os.waitpid with the the pid set to -1, this will wait for all the subprocess of the current process until they finish:

import os
import sys
import subprocess


proc = subprocess.Popen([sys.executable,
                         '-c',
                         'import subprocess;'
                         'subprocess.Popen("sleep 5", shell=True).wait()'])

pid, status = os.waitpid(-1, 0)

print pid, status

This is the result of pstree <pid> of different subprocess forked:

python───python───sh───sleep

Hope this can help :)