How to return stdout from long running process with subprocess and Popen?

Question

I'm using a pretty basic setup with subprocess.Popen() and directing stdout to a variable which I later return to a different part of my python script.

Here is my basic Popen code:

process = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
# wait for the process to terminate
out, err = process.communicate()
errcode = process.returncode
print out

This works great for many basic use cases like ls -al or similar. However, I am wondering how to handle getting output regularly and consistently from a longer (or indefinitely) running process such as tail -f foo.log. Is there a way to periodically read stdout in a loop? Or spawn a thread to check and return each on periodically? What is the best approach here?

Thanks!

Answer 1

I think it's important to note that the original code is not correct (or, rather, not safe). It will usually work, but nothing in your given example waits for the process to exit. It might still be running.

process.poll() and process.wait() are the two good options for this purpose.

---

When you don't know how large the output might be, communicate is dangerous because it buffers output into memory, and could run you out of memory. However, if you're using subprocess.PIPE, this is probably happening anyway.

You should carefully choose the targets for stdout and stderr depending on your needs. If it could be very large, writing to a file on disk might be the best option. That's a separate discussion, however.

---

To look at output without waiting on the process to close, you should run something like this in a separate thread:

while process.returncode is None:
    # handle output by direct access to stdout and stderr
    for line in process.stdout:
        print line
    # set returncode if the process has exited
    process.poll()

I'm open to comments about how you should actually access the file objects stdout and stderr, but this is what came to mind offhand.

Although this is the most robust way to handle spawned subprocesses, think hard about using process.wait() if you can -- it just makes everything much simpler.