Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

subprocess stdin buffer not flushing on newline with bufsize=1

I have two small python files, the first reads a line using input and then prints another line

a = input()
print('complete')

The second attempts to run this as a subprocess

import subprocess

proc = subprocess.Popen('./simp.py',
                        stdout=subprocess.PIPE,
                        stdin=subprocess.PIPE,
                        bufsize=1)
print('writing')
proc.stdin.write(b'hey\n')
print('reading')
proc.stdout.readline()

The above script will print "writing" then "reading" but then hang. At first I thought this was a stdout buffering issue, so I changed bufsize=1 to bufsize=0, and this does fix the problem. However, it seems it's the stdin that's causing the problem.

With bufsize=1, if I add proc.stdin.flush() below the write, the process continues. Both of these approaches seem clumsy since (1) unbuffered streams are slow (2) adding flushes everywhere is error-prone. Why does the above write not flush on a newline? The docs say that bufsize is used when creating stdin, stdout, and stderr stream for the subprocess, so what's causing the write to not flush on the newline?

like image 421
Ryan Haining Avatar asked Dec 19 '14 17:12

Ryan Haining


People also ask

How do I capture the output of a subprocess run?

To capture the output of the subprocess. run method, use an additional argument named “capture_output=True”. You can individually access stdout and stderr values by using “output. stdout” and “output.

What does subprocess Popen return?

Popen Function The function should return a pointer to a stream that may be used to read from or write to the pipe while also creating a pipe between the calling application and the executed command. Immediately after starting, the Popen function returns data, and it does not wait for the subprocess to finish.

What is Popen in Python?

Python method popen() opens a pipe to or from command. The return value is an open file object connected to the pipe, which can be read or written depending on whether mode is 'r' (default) or 'w'. The bufsize argument has the same meaning as in open() function.

What is stdin subprocess?

By default, subprocess. run() takes stdin (standard input) from our Python program and passes it through unchanged to the subprocess. For example, on a Linux or macOS system, the cat - command outputs exactly what it receives from stdin .


1 Answers

From the docs: "1 means line buffered (only usable if universal_newlines=True i.e., in a text mode)". This works:

import subprocess

proc = subprocess.Popen('./simp.py',
                        stdout=subprocess.PIPE,
                        stdin=subprocess.PIPE,
                        bufsize=1,
                        universal_newlines=True)

print('writing')
proc.stdin.write('hey\n')
print('reading')
proc.stdout.readline()
like image 108
tdelaney Avatar answered Oct 10 '22 17:10

tdelaney