process.stdout.readline() hangs. How to use it properly?

Question

I want to repeatedly send requests to process standard input and receive responses from standard output without calling subprocess multiple times. I can achieve a one-time request-response iteration using p.communicate however not to call the subprocess multiple times I need to use: process.stdout.readline() which hangs. How to use it properly? I use Python 2.7 64 bit, Windows 7. Thanks in advance.

main.py:

import subprocess
p = subprocess.Popen(['python','subproc.py'],stdin=subprocess.PIPE,stdout=subprocess.PIPE)

while True:
    s=raw_input('Enter message:')
    p.stdin.write(s)
    p.stdin.flush()
    response = p.stdout.readline()
    if response!= '':
        print "Process response:", response
    else:
        break

subproc.py:

from __future__ import division
import pyximport
s=raw_input()
print 'Input=',s

Answer 1

There are a couple of minor tweaks you can make to get this working. First is to disable buffered output in the child using the -u option. Second is to send a newline character along with the user-inputted message to the child process, so that the raw_input call in the child completes.

main.py

import subprocess

# We use the -u option to tell Python to use unbuffered output
p = subprocess.Popen(['python','-u', 'subproc.py'],
                     stdin=subprocess.PIPE,
                     stdout=subprocess.PIPE)

while True:
    s = raw_input('Enter message:')
    p.stdin.write(s + "\n")  # Include '\n'
    p.stdin.flush()
    response = p.stdout.readline()
    if response != '': 
        print "Process response:", response
    else:
        break

You should also wrap the child process in an infinite loop, or things will break after the first message is sent:

subproc.py:

while True:
    s = raw_input()
    print 'Input=',s

Output:

dan@dantop:~$ ./main.py 
Enter message:asdf
Process response: Input= asdf

Enter message:asdf
Process response: Input= asdf

Enter message:blah blah
Process response: Input= blah blah

Enter message:ok
Process response: Input= ok

Answer 2

It is not safe to assume that the child process will immediately get the complete data you send to its stdin, as buffers can get in the way. If you must hold the file open for further output then you should at least invoke its flush() method.

Furthermore, it is not safe to assume that the child process's output will be immmediately available to you to read. If it does not flush (or close) its output stream then the EOL could be buffered, and if you do nothing to prompt the child process to act further, then your readline() may wait forever. But your program then CAN'T do anything, because it's stuck in readline(). If the child process is built for this then you might get it to work, but otherwise you need to use a safer method, such as subprocess.communicate().

As you observe, it does not work to call communicate() more than once on the same subprocess. That's as you should expect from its documentation: it reads all output until end-of-file, and waits for the subprocess to terminate. To send multiple inputs in this mode, build a string containing all of them, pass it to communicate() and then read all the answers.

Alternatively, if you really need to alternate between writing and reading, and your subprocess is not specifically tooled for that, then it is safer to do the reading and writing in separate threads, without any assumption of the writes and reads interlacing perfectly.