What is the best way to run multiple subprocesses via fork()?

Question

A python script need to spawn multiple sub-processes via fork(). All of those child processes should run simultaneously and the parent process should be waiting for all of them to finish. Having an ability to set some timeout on a "slow" child would be nice. The parent process goes on processing the rest of the script after all kids are collected.

What is the best way to work it out? Thanks.

Answer 1

Simple example:

import os
chidren = []
for job in jobs:
    child = os.fork()
    if child:
        children.append(child)
    else:
        pass  # really should exec the job
for child in children:
    os.waitpid(child, 0)

Timing out a slow child is a little more work; you can use wait instead of waitpid, and cull the returned values from the list of children, instead of waiting on each one in turn (as here). If you set up an alarm with a SIGALRM handler, you can terminate the waiting after a specified delay. This is all standard UNIX stuff, not Python-specific...

Answer 2

Ephemient: each child in your code will stay in the for loop after his job ends. He will fork again and again. Moreover, the children that start when children[] is not empty will try to wait for some of their brothers at the end of the loop. Eventually someone will crash. This is a workaround:

import os, time

def doTheJob(job):
    for i in xrange(10):
        print job, i
        time.sleep(0.01*ord(os.urandom(1)))
        # random.random() would be the same for each process

jobs = ["A", "B", "C", "D", "E", "F", "G", "H", "I", "J"]
imTheFather = True
children = []

for job in jobs:
    child = os.fork()
    if child:
        children.append(child)
    else:
        imTheFather = False
        doTheJob(job)
        break

# in the meanwhile 
# ps aux|grep python|grep -v grep|wc -l == 11 == 10 children + the father

if imTheFather:
    for child in children:
        os.waitpid(child, 0)