comunicate between 2 processes with stdin and stdout

Question

I would like to write a simple script (A) that executes an external script (B)

• A should communicate with B by writing to its stdin and reading its stdout
• B should read its stdin and print it

all this should be done without closing the stream

A.py

import subprocess
process = subprocess.Popen(['python', 'B.py'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
for _ in range(3):
    process.stdin.write(b'hello')
    print(process.stdout.read())

B.py

import sys

for line in sys.stdin:
    print(line)

the output should be:

>>> b'hello'
>>> b'hello'
>>> b'hello'

The problem is that A just waits in

print(process.stdout.read())

if I modify A, by adding close():

for _ in range(3):
    process.stdin.write(b'hello')
    process.stdin.close()
    print(process.stdout.read())

I get:

>>> b'hello\n'
>>> Traceback (most recent call last):
>>>   File "A.py", line 7, in <module>
>>>     process.stdin.write(b'hello')
>>> ValueError: write to closed file

Answer 1

Use communicate()

Python already has communicate() method implemented (it goes to A.py, B.py is fine). However it's just suitable for simple communication (you know what data are you going to send up front), if you need a more complex one like:

send data to process B
read stdout
if stdout ...
    do something bases on stdout
    write to stdin

You have to implement your own communicate(), original implementation here.

---

Step by step

I've tested and debugged this step by step and here is what happens:

# For Popen(bufsize!=0)
A: process.stdin.write(b'hello\r\n')
B: line = sys.stdin.readline() # Hangs

So after adding bufsize=0 (unbuffered)

# Popen(bufsize=0)
A: process.stdin.write(b'hello\r\n') # Without \r\n B still hangs
B: line = sys.stdin.readline()
B: print('Send back', line.strip()) # Without strip it prints empty line
A: process.stdout.readline() # Hangs

So what works?

# Popen(bufsize=0)
A: process.stdin.write(b'hello\r\n')
B: line = sys.stdin.readline()
B: print('Send back', line.strip())
B: sys.stdout.flush()
A: process.stdout.readline()

---

Explained

You have buffering set to io.DEFAULT_BUFFER_SIZE (it's usually 4090B). From docs:

bufsize will be supplied as the corresponding argument to the io.open() function when creating the stdin/stdout/stderr pipe file objects: 0 means unbuffered (read and write are one system call and can return short), 1 means line buffered, any other positive value means use a buffer of approximately that size. A negative bufsize (the default) means the system default of io.DEFAULT_BUFFER_SIZE will be used.

So at first A won't flush because it hasn't filled its buffer yet and therefore B is waiting. It's not possible to simply process.stdin.flush() under Windows, so you have to use bufsize=0.

Also writing os.linesep (\r\n) is important because of readline().

Note: I believe it should have worked also with bufsize=1 (line buffering) but it didn't. I have no idea why.

Then the same happens in B when it won't flush sys.stdout and it surprises me, that B:sys.stdout is not set to unbuffered because:

bufsize will be supplied as the corresponding argument to the io.open() function when creating the stdin/stdout/stderr pipe file objects

Anyway, you have to call sys.stdout.flush() in B.

It works with close() because it forces flush().

---

Gimme teh codes

A.py:

import subprocess
import sys

process = subprocess.Popen([sys.executable, r'B.py'], stdin=subprocess.PIPE, 
                            stdout=subprocess.PIPE, bufsize=0)
for _ in range(3):
    process.stdin.write(b'hello\r\n')
    print(process.stdout.readline())

B.py:

import sys

for line in sys.stdin:
    print('Send back', line.strip())
    sys.stdout.flush()