running several system commands in parallel in Python

Question

I write a simple script that executes a system command on a sequence of files. To speed things up, I'd like to run them in parallel, but not all at once - i need to control maximum number of simultaneously running commands. What whould be the easiest way to approach this ?

Answer 1

If you are calling subprocesses anyway, I don't see the need to use a thread pool. A basic implementation using the subprocess module would be

import subprocess import os import time  files = <list of file names> command = "/bin/touch" processes = set() max_processes = 5  for name in files:     processes.add(subprocess.Popen([command, name]))     if len(processes) >= max_processes:         os.wait()         processes.difference_update([             p for p in processes if p.poll() is not None]) 

On Windows, os.wait() is not available (nor any other method of waiting for any child process to terminate). You can work around this by polling in certain intervals:

for name in files:     processes.add(subprocess.Popen([command, name]))     while len(processes) >= max_processes:         time.sleep(.1)         processes.difference_update([             p for p in processes if p.poll() is not None]) 

The time to sleep for depends on the expected execution time of the subprocesses.

Answer 2

The answer from Sven Marnach is almost right, but there is a problem. If one of the last max_processes processes ends, the main program will try to start another process, and the for looping will end. This will close the main process, which can in turn close the child processes. For me, this behavior happened with the screen command.

The code in Linux will be like this (and will only work on python2.7):

import subprocess import os import time  files = <list of file names> command = "/bin/touch" processes = set() max_processes = 5  for name in files:     processes.add(subprocess.Popen([command, name]))     if len(processes) >= max_processes:         os.wait()         processes.difference_update(             [p for p in processes if p.poll() is not None]) #Check if all the child processes were closed for p in processes:     if p.poll() is None:         p.wait()