How do I get 'real-time' information back from a subprocess.Popen in python (2.5)

Question

I'd like to use the subprocess module in the following way:

1. create a new process that potentially takes a long time to execute.
2. capture stdout (or stderr, or potentially both, either together or separately)
3. Process data from the subprocess as it comes in, perhaps firing events on every line received (in wxPython say) or simply printing them out for now.

I've created processes with Popen, but if I use communicate() the data comes at me all at once, once the process has terminated.

If I create a separate thread that does a blocking readline() of myprocess.stdout (using stdout = subprocess.PIPE) I don't get any lines with this method either, until the process terminates. (no matter what I set as bufsize)

Is there a way to deal with this that isn't horrendous, and works well on multiple platforms?

Answer 1

Update with code that appears not to work (on windows anyway)

class ThreadWorker(threading.Thread):     def __init__(self, callable, *args, **kwargs):         super(ThreadWorker, self).__init__()         self.callable = callable         self.args = args         self.kwargs = kwargs         self.setDaemon(True)      def run(self):         try:             self.callable(*self.args, **self.kwargs)         except wx.PyDeadObjectError:             pass         except Exception, e:             print e    if __name__ == "__main__":     import os     from subprocess import Popen, PIPE      def worker(pipe):         while True:             line = pipe.readline()             if line == '': break             else: print line      proc = Popen("python subprocess_test.py", shell=True, stdin=PIPE, stdout=PIPE, stderr=PIPE)      stdout_worker = ThreadWorker(worker, proc.stdout)     stderr_worker = ThreadWorker(worker, proc.stderr)     stdout_worker.start()     stderr_worker.start()     while True: pass 

Answer 2

stdout will be buffered - so you won't get anything till that buffer is filled, or the subprocess exits.

You can try flushing stdout from the sub-process, or using stderr, or changing stdout on non-buffered mode.