python multiprocessing.Process.terminate - How to kill child processes

Question

This code:

import multiprocessing as mp
from threading import Thread
import subprocess
import time

class WorkerProcess(mp.Process):
  def run(self):
      # Simulate long running task
      self.subprocess = subprocess.Popen(['python', '-c', 'import time; time.sleep(1000)'])
      self.code = self.subprocess.wait()


class ControlThread(Thread):
  def run():
      jobs = []
      for _ in range(2):
          job = WorkerProcess()
          jobs.append(job)
          job.start()

      # wait for a while and then kill jobs
      time.sleep(2)
      for job in jobs:
          job.terminate()

if __name__ == "__main__":
    controller = ControlThread()
    controller.start()

When I terminate the spawned WorkerProcess instances. They die just fine, however the subprocesses python -c 'import time; time.sleep(1000) runs until completition. This is well documented in the official docs, but how do I kill the child processes of a killed process?

A possbile soultion might be:

• Wrap WorkerProcess.run() method inside try/except block catching SIGTERM, and terminating the subprocess.call call. But I am not sure how to catch the SIGTERM in the WorkerProcess

• I also tried setting signal.signal(signal.SIGINT, handler) in the WorkerProcess, but I am getting ValuError, because it is allowed to be set only in the main thread.

What do I do now?

Answer 1

EDIT: As @svalorzen pointed out in comments this doesn't really work since the reference to self.subprocess is lost.

---

Finally came to a clean, acceptable solution. Since mp.Process.terminate is a method, we can override it.

class WorkerProcess(mp.Process):
    def run(self):
        # Simulate long running task
        self.subprocess = subprocess.Popen(['python', '-c', 'import time; time.sleep(1000)'])
        self.code = self.subprocess.wait()

    # HERE
    def terminate(self):
      self.subprocess.terminate()
      super(WorkerProcess, self).terminate()

Answer 2

You can use queues to message to your subprocesses and ask them nicely to terminate their children before exiting themselves. You can't use signals in anywhere else but your main thread, so signals are not suitable for this.

Curiously, when I modify the code like this, even if I interrupt it with control+C, subprocesses will die as well. This may be OS related thing, though.

import multiprocessing as mp
from threading import Thread
import subprocess
import time
from Queue import Empty


class WorkerProcess(mp.Process):
    def __init__(self,que):
        super(WorkerProcess,self).__init__()
        self.queue = que

    def run(self):
        # Simulate long running task
        self.subprocess = subprocess.Popen(['python', '-c', 'import time; time.sleep(1000)'])
        while True:
            a = self.subprocess.poll()
            if a is None:
                time.sleep(1)
                try:
                    if self.queue.get(0) == "exit":
                        print "kill"
                        self.subprocess.kill()
                        self.subprocess.wait()
                        break
                    else:
                        pass
                except Empty:
                    pass
                print "run"
            else:
                print "exiting"


class ControlThread(Thread):

    def run(self):
        jobs = []
        queues = []
        for _ in range(2):
            q = mp.Queue()
            job = WorkerProcess(q)
            queues.append(q)
            jobs.append(job)
            job.start()

        # wait for a while and then kill jobs
        time.sleep(5)
        for q in queues:
            q.put("exit")
        time.sleep(30)

if __name__ == "__main__":
    controller = ControlThread()
    controller.start()

Hope this helps.

Hannu