Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Reading/writing to a Popen() subprocess

I'm trying to talk to a child process using the python subprocess.Popen() call. In my real code, I'm implementing a type of IPC, so I want to write some data, read the response, write some more data, read the response, and so on. Because of this, I cannot use Popen.communicate(), which otherwise works well for the simple case.

This code shows my problem. It never even gets the first response, hangs at the first "Reading result". Why? How can I make this work as I expect?

import subprocess
p = subprocess.Popen(["sed", 's/a/x/g'],
                     stdout = subprocess.PIPE,
                     stdin = subprocess.PIPE)

p.stdin.write("abc\n")
print "Reading result:"
print p.stdout.readline()

p.stdin.write("cat\n")
print "Reading result:"
print p.stdout.readline()
like image 957
Mats Ekberg Avatar asked Apr 28 '12 13:04

Mats Ekberg


People also ask

How do I write to a Python subprocess stdin?

To write to a Python subprocess' stdin, we can use the communicate method. to call Popen with the command we want to run in a list. And we set stdout , stdin , and stderr all to PIPE to pipe them to their default locations.

How do I capture the output of a subprocess run?

To capture the output of the subprocess. run method, use an additional argument named “capture_output=True”. You can individually access stdout and stderr values by using “output. stdout” and “output.

What is subprocess popen ()?

The subprocess module defines one class, Popen and a few wrapper functions that use that class. The constructor for Popen takes arguments to set up the new process so the parent can communicate with it via pipes. It provides all of the functionality of the other modules and functions it replaces, and more.


2 Answers

sed's output is buffered and only outputs its data until enough has been cumulated or the input stream is exhausted and closed.

Try this:

import subprocess
p = subprocess.Popen(["sed", 's/a/x/g'],
                     stdout = subprocess.PIPE,
                     stdin = subprocess.PIPE)

p.stdin.write("abc\n")
p.stdin.write("cat\n")
p.stdin.close()

print "Reading result 1:"
print p.stdout.readline()

print "Reading result 2:"
print p.stdout.readline()

Be aware that this cannot be done reliably which huge data as wriring to stdin blocks once the buffer is full. The best way to do is using communicate().

like image 87
glglgl Avatar answered Sep 18 '22 06:09

glglgl


I would try to use Popen().communicate() if you can as it does a lot of nice things for you, but if you need to use Popen() exactly as you described, you'll need to set sed to flush its buffer after newlines with the -l option:

p = subprocess.Popen(['sed', '-l', 's/a/x/g'],
                     stdout=subprocess.PIPE,
                     stdin=subprocess.PIPE)

and your code should work fine

like image 35
gnr Avatar answered Sep 19 '22 06:09

gnr