How does a python process exit gracefully after receiving SIGTERM while waiting on a semaphore?

Question

I have a Python process which spawns 5 other Python processes using the multiprocessing module. Let's call the parent process P0 and the others P1-P5. The requirement is, if we send a SIGTERM to P0, it should shut down P1 to P5 first and then exit itself.

The catch is P1 and P5 are waiting on semaphores. So when I send SIGTERM to these processes, they invoke the signal handler and exit. But since they are waiting on semaphore, they throw an exception. Is there any way to catch that exception before exit, so that P0 to P5 can make a graceful exit?

Traceback:

Traceback (most recent call last):
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 258, in _bootstrap
Traceback (most recent call last):
Process Process-2:
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 258, in _bootstrap
Traceback (most recent call last):
self.run()
File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
Process Process-5:
Traceback (most recent call last):
File "/usr/lib64/python2.7/multiprocessing/process.py", line 258, in _bootstrap
  self.run()
File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run
  self._target(*self._args, **self._kwargs)
File "/opt/fireeye/scripts/mip/StaticAnalysisRunner.py", line 45, in run
  qsem.acquire()

Answer 1

You can install a signal handler which throws an exception which is then caught in the subprocess to handle exits gracefully.

Here is an example of a script which waits in a semaphore in a subprocess and terminates gracefully when sent a SIGTERM.

#!/usr/bin/env python

import signal
import time
import multiprocessing

class GracefulExit(Exception):
    pass


def signal_handler(signum, frame):
    raise GracefulExit()


def subprocess_function():
    try:
        sem = multiprocessing.Semaphore()
        print "Acquiring semaphore"
        sem.acquire()
        print "Semaphore acquired"

        print "Blocking on semaphore - waiting for SIGTERM"
        sem.acquire()
    except GracefulExit:
        print "Subprocess exiting gracefully"


if __name__ == "__main__":

    # Use signal handler to throw exception which can be caught to allow
    # graceful exit.
    signal.signal(signal.SIGTERM, signal_handler)

    # Start a subprocess and wait for it to terminate.
    p = multiprocessing.Process(target=subprocess_function)
    p.start()

    print "Subprocess pid: %d" % p.pid

    p.join()

An example run of this script is as follows:

$ ./test.py 
Subprocess pid: 7546
Acquiring semaphore
Semaphore acquired
Blocking on semaphore - waiting for SIGTERM
----> Use another shell to kill -TERM 7546
Subprocess exiting gracefully

There is no traceback from the subprocess and the flow shows that the subprocess exits in a graceful manner. This is because the SIGTERM is caught by the subprocess signal handler which throws a normal Python exception which can be handled inside the process.