Python losing control of subprocess?

Question

I'm using a commercial application that uses Python as part of its scripting API. One of the functions provided is something called App.run(). When this function is called, it starts a new Java process that does the rest of the execution. (Unfortunately, I don't really know what it's doing under the hood as the supplied Python modules are .pyc files, and many of the Python functions are SWIG generated).

The trouble I'm having is that I'm building the App.run() call into a larger Python application that needs to do some guaranteed cleanup code (closing a database, etc.). Unfortunately, if the subprocess is interrupted with Ctrl+C, it aborts and returns to the command line without returning control to the main Python program. Thus, my cleanup code never executes.

So far I've tried:

1. Registering a function with atexit... doesn't work
2. Putting cleanup in a class __del__ destructor... doesn't work. (App.run() is inside the class)
3. Creating a signal handler for Ctrl+C in the main Python app... doesn't work
4. Putting App.run() in a Thread... results in a Memory Fault after the Ctrl+C
5. Putting App.run() in a Process (from multiprocessing)... doesn't work

Any ideas what could be happening?

Answer 1

This is just an outline- but something like this?

import os

cpid = os.fork()
if not cpid:
    # change stdio handles etc
    os.setsid() # Probably not needed
    App.run()
    os._exit(0)

os.waitpid(cpid)
# clean up here

(os.fork is *nix only)

The same idea could be implemented with subprocess in an OS agnostic way. The idea is running App.run() in a child process and then waiting for the child process to exit; regardless of how the child process died. On posix, you could also trap for SIGCHLD (Child process death). I'm not a windows guru, so if applicable and subprocess doesn't work, someone else will have to chime in here.

After App.run() is called, I'd be curious what the process tree looks like. It's possible its running an exec and taking over the python process space. If thats happening, creating a child process is the only way I can think of trapping it.

Answer 2

If try: App.run() finally: cleanup() doesn't work; you could try to run it in a subprocess:

import sys
from subprocess import call

rc = call([sys.executable, 'path/to/run_app.py'])
cleanup()

Or if you have the code in a string you could use -c option e.g.:

rc = call([sys.executable, '-c', '''import sys
print(sys.argv)
'''])

You could implement @tMC's suggestion using subprocess by adding preexec_fn=os.setsid argument (note: no ()) though I don't see how creating a process group might help here. Or you could try shell=True argument to run it in a separate shell.

You might give another try to multiprocessing:

import multiprocessing as mp

if __name__=="__main__":
   p = mp.Process(target=App.run)
   p.start()
   p.join()
   cleanup()