python, subprocess: reading output from subprocess

Question

I have following script:

#!/usr/bin/python

while True:
    x = raw_input()
    print x[::-1]

I am calling it from ipython:

In [5]: p = Popen('./script.py', stdin=PIPE)

In [6]: p.stdin.write('abc\n')
cba

and it works fine.

However, when I do this:

In [7]: p = Popen('./script.py', stdin=PIPE, stdout=PIPE)

In [8]: p.stdin.write('abc\n')

In [9]: p.stdout.read()

the interpreter hangs. What am I doing wrong? I would like to be able to both write and read from another process multiple times, to pass some tasks to this process. What do I need to do differently?

EDIT 1

If I use communicate, I get this:

In [7]: p = Popen('./script.py', stdin=PIPE, stdout=PIPE)

In [8]: p.communicate('abc\n')
Traceback (most recent call last):
  File "./script.py", line 4, in <module>
    x = raw_input()
EOFError: EOF when reading a line
Out[8]: ('cba\n', None)

EDIT 2

I tried flushing:

#!/usr/bin/python

import sys

while True:
        x = raw_input()
        print x[::-1]
        sys.stdout.flush()

and here:

In [5]: from subprocess import PIPE, Popen

In [6]: p = Popen('./script.py', stdin=PIPE, stdout=PIPE)

In [7]: p.stdin.write('abc')

In [8]: p.stdin.flush()

In [9]: p.stdout.read()

but it hangs again.

Answer 1

I believe there are two problems at work here:

1) Your parent script calls p.stdout.read(), which will read all data until end-of-file. However, your child script runs in an infinite loop so end-of-file will never happen. Probably you want p.stdout.readline()?

2) In interactive mode, most programs do buffer only one line at a time. When run from another program, they buffer much more. The buffering improves efficiency in many cases, but causes problems when two programs need to communicate interactively.

After p.stdin.write('abc\n') add:

p.stdin.flush()

In your subprocess script, after print x[::-1] add the following within the loop:

sys.stdout.flush()

(and import sys at the top)

Answer 2

The subprocess method check_output can be useful for this:

output = subprocess.check_output('./script.py')

And output will be the stdout from the process. If you need stderr, too:

output = subprocess.check_output('./script.py', stderr=subprocess.STDOUT)

Because you avoid managing pipes directly, it may circumvent your issue.