python subprocess with shell=True: redirections and platform-independent subprocess killing

Question

I'm having a hard time getting what I want out of the python subprocess module (which supposed to an unified/platform-independent abstraction, afaik, but don't get me started on that :)).

So the simple thing what I'm after is the following. I want to

• Launch an external (stdio) application (possibly with subprocess), where I use shell-style redirections (like './myapp >stdout_log >stderr_log')
  • Basically i want to execute a shell command-line, so I have to specify shell=True for subprocess.Popen() (or else redirections in the command-line won't work)
• I want to launch this command line in an async fashion (so it runs as an independent sub-process, but my python process won't wait for it's completion)
• (My parent python process would look at the child process's logs from time to time to extract information, but this is irrelevant to the question)
• If my parent python process decides, it should be able to terminate this child process.

Now, my main problems are that

• I'm basically forced to use shell=True, to get redirections to work
• Processing the child's stdout/stderr in the parent python process is not an option, since I couldn't find functionality for doing it in a non-waiting way (and the parent python process must do other things while the child is running)
• If I use shell=True then subprocess.kill() will only terminate the shell but not the child process
• I'd need a reliable child process termination method that works on any platform (but at least linux and windows)

I hope I was specific enough. Thanks for any tips/hints in advance -- I just spent a whole day with subprocess, and IMHO it's a pain far from either platform-independent or simple :( (but perhaps it's just me)

UPDATE (2010-10-13):

If you launch a sub-process (even with shell=False), then the subprocess.Popen.kill() function will only kill that sub-process (so if there are any "grandchild" processes, they won't be terminated.)

I read about using the preexec_fn parameter to set the sid on all child processes, but it's unix-only: timeout a subprocess

Answer 1

Last time I was in a similar situation, I found out the easiest (and practically the only) solution was to kick off a thread which takes care of your child process. You can take different routes with this method, be it to parse the piping of the shell-style command and perform those in python code (which you said wasn't an option due to the blocking), which would at the same time fix your killing problem. Basically, thread encapsulation seems like the way to go.

Sadly my experience with subprocess is all on the Windows platform, which has tons of its own little quirks. subprocess has a lot of flaws all-around it seems, although it must do an okay job given the existence of the popen, popen2 and so forth modules that it is supposed to replace.