How to avoid the deadlock in a subprocess without using communicate()

Question

proc = subprocess.Popen(['start'],stdin=subprocess.PIPE,stdout=subprocess.PIPE)
proc.stdin.write('issue commands')
proc.stdin.write('issue more commands')
output = proc.stdout.read()   # Deadlocked here

# Actually I have more commands to issue here 

I know that communicate() can give me a solution to this problem but I wanted to issue more commands later. But communicate() kind of closes the subprocess.

Is there any know work around here for this. I am trying to interact with a router using a python wrapper. So I have more commands coming up which also could produce some output. In that situation how can I read without terminating the subprocess.

Answer 1

output = proc.stdout.read()   # Deadlocked here

That line causes a deadlock because read() won't return until it reads EOF, which is sent when the other side closes its stdout (e.g. when the subprocess terminates). Instead, you want to read line oriented input:

output = proc.stdout.readline()

readline() will return after a newline (or EOF) is read i.e. after readline() reads a line.

Your next deadlock will result from either:

1. Not adding a newline to the output you send to a subprocess--when the subprocess is trying to read line oriented input, i.e. the subprocess is looking for a newline while reading from stdin.

2. Not flushing the output, which means the other side never sees any data to read, so the other side hangs waiting for data.

For efficiency, when you write to a file (which includes stdout,stdin) output is buffered, which means instead of actually writing the output to a file, python tricks you and stores the output in a list (known as a buffer). Then when the list grows to a certain size, python writes all the output to the file at once, which is more efficient than writing a line at a time.

Not adding a newline to all the output that you send to a subprocess can be corrected easily enough; however, discovering all the places where you need to flush buffers is more difficult. Here's an example:

prog.py:

#!/usr/bin/env python3.4 -u

import sys

print('What is your name?') 
name = input()
print(name)

print("What is your number?")
number = input()
print(number)

Suppose you want to drive that program with another program?

1. Make prog.py executable: $ chmod a+x prog.py

2. Note the shebang line.

3. Note the -u flag for the python interpreter in the shebang line, which means all the output for that program will be unbuffered, i.e. it will be written directly to stdout--not accumulated in a buffer.

---

import subprocess as sp

child = sp.Popen(
    ['./prog.py'],
    stdin = sp.PIPE,
    stdout = sp.PIPE
)

print_question = child.stdout.readline().decode('utf-8')  #PIPE's send a bytes type, 
                                                          #so convert to str type
name = input(print_question)
name = "{}\n".format(name)
child.stdin.write(
    name.encode('utf-8') #convert to bytes type
)
child.stdin.flush()

print_name = child.stdout.readline().decode('utf-8')
print("From client: {}".format(name))

print_question = child.stdout.readline().decode('utf-8')

number = input(print_question)
number = "{}\n".format(number)
child.stdin.write(
    number.encode('utf-8')
)
child.stdin.flush()

print_number = child.stdout.readline().decode('utf-8')
print("From client: {}".format(print_number))

Response to comment:

You could be suffering from buffering. Most programs buffer output for efficiency. Furthermore, some programs will try to determine if their stdout is connected to a terminal--if it is, then the program employs line buffering, which means that output is retrieved from the buffer and actually written to stdout every time the program outputs a newline. That allows a human using the connected terminal to interact with the program.

On the other hand, if the program’s stdout is not connected to a terminal, the program will block buffer, which means output is retrieved from the buffer and actually written to stdout only after the buffer has grown to a specific size, say 4K of data. If the program is block buffering and it outputs less than 4K, then nothing actually gets written to stdout. Block buffering allows other computer programs to retrieve the output of the program with better efficiency.

If the router program is block buffering, and it outputs less than the block size, then nothing actually gets written to stdout, so there is nothing for your python program to read.

Your python program is not a terminal, so the router program may be block buffering. As J.F. Sebastian pointed out in the comments, there are ways to trick the router program into thinking your python program is a terminal, which will cause the router program to line buffer, and therefore you will be able to read from its stdout with readline(). Check out pexpect.