In Python, how do I know when a process is finished?

Question

From within a Python GUI (PyGTK) I start a process (using multiprocessing). The process takes a long time (~20 minutes) to finish. When the process is finished I would like to clean it up (extract the results and join the process). How do I know when the process has finished?

My colleague suggested a busy loop within the parent process that checks if the child process has finished. Surely there is a better way.

In Unix, when a process is forked, a signal handler is called from within the parent process when the child process has finished. But I cannot see anything like that in Python. Am I missing something?

How is it that the end of a child process can be observed from within the parent process? (Of course, I do not want to call Process.join() as it would freeze up the GUI interface.)

This question is not limited to multi-processing: I have exactly the same problem with multi-threading.

Answer 1

I think as a part of making python multi-platform, simple things like SIGCHLD must be done yourself. Agreed, this is a little more work when all you want to do is know when the child is done, but it really isn't THAT painful. Consider the following that uses a child process to do the work, two multiprocessing.Event instances, and a thread to check if the child process is done:

import threading from multiprocessing import Process, Event from time import sleep  def childsPlay(event):     print "Child started"     for i in range(3):         print "Child is playing..."         sleep(1)     print "Child done"     event.set()  def checkChild(event, killEvent):     event.wait()     print "Child checked, and is done playing"     if raw_input("Do again? y/n:") == "y":         event.clear()         t = threading.Thread(target=checkChild, args=(event, killEvent))         t.start()         p = Process(target=childsPlay, args=(event,))         p.start()     else:         cleanChild()         killEvent.set()  def cleanChild():     print "Cleaning up the child..."  if __name__ == '__main__':     event = Event()     killEvent = Event()      # process to do work     p = Process(target=childsPlay, args=(event,))     p.start()      # thread to check on child process     t = threading.Thread(target=checkChild, args=(event, killEvent))     t.start()      try:         while not killEvent.is_set():             print "GUI running..."             sleep(1)     except KeyboardInterrupt:         print "Quitting..."         exit(0)     finally:         print "Main done" 

EDIT

Joining to all processes and threads created is a good practice because it will help indicate when zombie (never-finishing) processes/threads are being created. I've altered the above code making a ChildChecker class that inherits from threading.Thread. It's sole purpose is to start a job in a separate process, wait for that process to finish, and then notify the GUI when everything is complete. Joining on the ChildChecker will also join the process it is "checking". Now, if the process doesn't join after 5 seconds, the thread will force terminate the process. Enter "y" creates starts a child process running "endlessChildsPlay" that must demonstrate force termination.

import threading from multiprocessing import Process, Event from time import sleep  def childsPlay(event):     print "Child started"     for i in range(3):         print "Child is playing..."         sleep(1)     print "Child done"     event.set()  def endlessChildsPlay(event):     print "Endless child started"     while True:         print "Endless child is playing..."         sleep(1)         event.set()     print "Endless child done"  class ChildChecker(threading.Thread):     def __init__(self, killEvent):         super(ChildChecker, self).__init__()         self.killEvent = killEvent         self.event = Event()         self.process = Process(target=childsPlay, args=(self.event,))      def run(self):         self.process.start()          while not self.killEvent.is_set():             self.event.wait()             print "Child checked, and is done playing"             if raw_input("Do again? y/n:") == "y":                 self.event.clear()                 self.process = Process(target=endlessChildsPlay, args=(self.event,))                 self.process.start()             else:                 self.cleanChild()                 self.killEvent.set()      def join(self):         print "Joining child process"         # Timeout on 5 seconds         self.process.join(5)          if self.process.is_alive():             print "Child did not join!  Killing.."             self.process.terminate()         print "Joining ChildChecker thread"         super(ChildChecker, self).join()       def cleanChild(self):         print "Cleaning up the child..."  if __name__ == '__main__':     killEvent = Event()     # thread to check on child process     t = ChildChecker(killEvent)     t.start()      try:         while not killEvent.is_set():             print "GUI running..."             sleep(1)     except KeyboardInterrupt:         print "Quitting..."         exit(0)     finally:         t.join()         print "Main done"