python subprocess module: looping over stdout of child process

Question

I have some commands which I am running using the subprocess module. I then want to loop over the lines of the output. The documentation says do not do data_stream.stdout.read which I am not but I may be doing something which calls that. I am looping over the output like this:

for line in data_stream.stdout:
   #do stuff here
   .
   .
   .

Can this cause deadlocks like reading from data_stream.stdout or are the Popen modules set up for this kind of looping such that it uses the communicate code but handles all the callings of it for you?

Answer 1

The two answer have caught the gist of the issue pretty well: don't mix writing something to the subprocess, reading something from it, writing again, etc -- the pipe's buffering means you're at risk of a deadlock. If you can, write everything you need to write to the subprocess FIRST, close that pipe, and only THEN read everything the subprocess has to say; communicate is nice for the purpose, IF the amount of data is not too large to fit in memory (if it is, you can still achieve the same effect "manually").

If you need finer-grain interaction, look instead at pexpect or, if you're on Windows, wexpect.

Answer 2

You have to worry about deadlocks if you're communicating with your subprocess, i.e. if you're writing to stdin as well as reading from stdout. Because these pipes may be cached, doing this kind of two-way communication is very much a no-no:

data_stream = Popen(mycmd, stdin=PIPE, stdout=PIPE)
data_stream.stdin.write("do something\n")
for line in data_stream:
  ...  # BAD!

However, if you've not set up stdin (or stderr) when constructing data_stream, you should be fine.

data_stream = Popen(mycmd, stdout=PIPE)
for line in data_stream.stdout:
   ...  # Fine

If you need two-way communication, use communicate.