Python join a process without blocking parent

Question

I'm writing a program that will watch a particular directory for new files containing download URLs. Once a new file is detected, it will create a new process to do the actual download while the parent continues to watch the directory. I'm using the Process interface from multiprocessing. The problem I have is that unless I call process.join() the child process is still running, but process.join() is a blocking function which defeats the purpose of creating the child to handle the actual download.

My question is, is there a way to join the child process in a non-blocking manner which will allow the parent to keep doing its thing?

Partial code:

def main(argv):   # parse command line args   ...   # set up variables   ...   watch_dir(watch_dir, download_dir)   def watch_dir(wDir, dDir):   # Grab the current watch directory listing   before = dict([(f, None) for f in os.listdir (wDir)])    # Loop FOREVER   while 1:     # sleep for 10 secs     time.sleep(10)      # Grab the current dir listing     after = dict([(f, None) for f in os.listdir (wDir)])      # Get the list of new files     added = [f for f in after if not f in before]     # Get the list of deleted files     removed = [f for f in before if not f in after]      if added:       # We have new files, do your stuff       print "Added: ", ", ".join(added)        # Call the new process for downloading       p = Process(target=child, args=(added, wDir, dDir))       p.start()       p.join()      if removed:       # tell the user the file was deleted       print "Removed: ", ", ".join(removed)      # Set before to the current     before = after  def child(filename, wDir, dDir):   # Open filename and extract the url   ...   # Download the file and to the dDir directory   ...   # Delete filename from the watch directory   ...   # exit cleanly   os._exit(0) 

The parent waits for the child to finish execution before continuing after p.join() which is (as far as I can tell) correct. But that defeats the whole purpose of creating the child. If I leave off p.join() then the child remains active and a ps ax | grep python give me 'python <defunct>'.

I'd like the child to finish up what its doing and go away without holding up the parent. Is there a way to do it?

Answer 1

You can set up a separate thread which does the joining. Have it listen on a queue into which you push the subprocess handles:

class Joiner(Thread):     def __init__(self, q):         self.__q = q     def run(self):         while True:             child = self.__q.get()             if child == None:                 return             child.join() 

Then, instead of p.join(), do joinq.put(p) and do a joinq.put(None) to signal the thread to stop. Make sure you use a FIFO queue.