subprocess popen.communicate() vs. stdin.write() and stdout.read()

Question

I have noticed two different behaviors with two approaches that should have result in the same outcome.

The goal - to execute an external program using subprocess module, send some data and read the results.

The external program is PLINK, platform is WindowsXP, Python version 3.3.

The main idea-

execution=["C:\\Pr..\\...\\plink.exe", "-l", username, "-pw", "***", IP]
a=subprocess.Popen(execution, bufsize=0, stdout=PIPE, stdin=PIPE, stderr=STDOUT, shell=False)
con=a.stdout.readline()
if (con.decode("utf-8").count("FATAL ERROR: Network error: Connection timed out")==0):
   a.stdin.write(b"con rout 1\n")
   print(a.stdout.readline().decode("utf-8"))
   a.stdin.write(b"infodf\n")
   print(a.stdout.readline().decode("utf-8"))
else:
   print("ERROR")
a.kill()

So far so good.

Now, I want to be able to do a loop (after each write to the sub process's stdin), that waits untill EOF of the sub process's stdout, print it, then another stdin command, and so on.

So I first tried what previous discussions about the same topic yield (live output from subprocess command, read subprocess stdout line by line, python, subprocess: reading output from subprocess) .

And it didnt work (it hangs forever) because the PLINK process is remaining alive untill I kill it myself, so there is no use of waiting for the stdout of the sub process to reach EOF or to do a loop while stdout is true because it will always be true until I kill it.

So I decided to read from stdout twice every time I am writing to stdin (good enought for me)-

execution=["C:\\Pr..\\...\\plink.exe", "-l", username, "-pw", "***", IP]
a=subprocess.Popen(execution, bufsize=0, stdout=PIPE, stdin=PIPE, stderr=STDOUT, shell=False)
con=a.stdout.readline()
if (con.decode("utf-8").count("FATAL ERROR: Network error: Connection timed out")==0):
   a.stdin.write(b"con rout 1\n")
   print(a.stdout.readline().decode("utf-8"))
   print(a.stdout.readline().decode("utf-8"))   //the extra line [1]
   a.stdin.write(b"infodf\n")
   print(a.stdout.readline().decode("utf-8"))
   print(a.stdout.readline().decode("utf-8"))   //the extra line [2]
else:
   print("ERROR")
a.kill()

But the first extra readline() hangs forever, as far as I understand, for the same reason I mentioned. The first extra readline() waits forever for output, because the only output was already read in the first readline(), and because PLINK is alive, the function just "sit" there and waits for a new output line to get.

So I tried this code, expecting the same hang because PLINK never dies until i kill it-

execution=["C:\\Pr..\\...\\plink.exe", "-l", username, "-pw", "***", IP]
a=subprocess.Popen(execution, bufsize=0, stdout=PIPE, stdin=PIPE, stderr=STDOUT, shell=False)
con=a.stdout.readline()
if (con.decode("utf-8").count("FATAL ERROR: Network error: Connection timed out")==0):
   a.stdin.write(b"con rout 1\n")
   print(a.stdout.readline().decode("utf-8"))
   a.stdin.write(b"infodf\n")
   print(a.stdout.readline().decode("utf-8"))
   print(a.communicate()[0].decode("utf-8"))     //Popen.communicate() function
else:
   print("ERROR")
a.kill()

I tried that because according to the documentation of communicate(), the function wait until the process is ended, and then it finishes. Also, it reads from stdout until EOF. (same as writing and reading stdout and stdin)

But communicate() finishes and does not hang, in opposite of the previous code block.

What am I missing here? why when using communicate() the PLINK ends, but when using readline() it does not?

Answer 1

Your program without communicate() deadlocks because both processes are waiting on each other to write something before they write anything more themselves.

communicate() does not deadlock in your example because it closes the stream, like the command a.stdin.close() would. This sends an EOF to your subprocess, letting it know that there is no more input coming, so it can close itself, which in turn closes its output, so a.stdout.read() eventually returns an EOF (empty string).

There is no special signal that your main process will receive from your subprocess to let you know that it is done writing the results from one command, but is ready for another command.

This means that to communicate back and forth with one subprocess like you're trying to, you must read the exact number of lines that the subprocess sends. Like you saw, if you try to read too many lines, you deadlock. You might be able to use what you know, such as the command you sent it, and the output you have seen so far, to figure out exactly how many lines to read.