piping together several subprocesses

Question

I have 5 processes p1,p2,...,p5 where I want to write some data to stdin of p1, pipe p1 output to p2 stdin and finally read the final result from output of p5.

What I have tried so far:

p1 = Popen(['p1'], stdin=PIPE, stdout=PIPE)
p2 = Popen(['p2'], stdin=p1.stdout, stdout=PIPE)
...
p5 = Popen(['p5'], stdin=p4.stdout, stdout=PIPE)

# write data to stdin
p1.stdin.write(indata)
p1.stdin.close()

# not sure in what order to close the pipes here, if at all

# read output
out = p5.stdout.read()
print out

The last code snipped just hangs, because I must be doing the read/write operations incorrectly.

I was able to get working single processes using communicate() and two processes without suppling any input to first one (example from Python docs):

output=`dmesg | grep hda`
==>
p1 = Popen(["dmesg"], stdout=PIPE)
p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
p1.stdout.close()  # Allow p1 to receive a SIGPIPE if p2 exits.
output = p2.communicate()[0]

But I cannot figure out, how to supply input to first process without hanging the interpreter.

I could also use a bash script for that (that I've already written and works), but I'd like to know, how to achieve same with Python.

So, I'd like to ask, how do do all that properly, specifically in what order to do the read/write/close operations on pipes?

I'm working on 64-bit Linux, if that matters.

EDIT: I forgot to mention that all processes p1,..p5 consume all input they are given, process it, write to stdout and then terminate. Thus, the processes next in pipeline should not terminate before the previous ones have finished processing.

EDIT2: I know that I could also use

command = 'bash -c "p1 | p2 | p3 | p4 | p5"'
proc = Popen([command], shell=True)
out, err = proc.communicate(input=indata)
print out

but my main interest is getting know how to chain the pipes purely in python code.

Answer 1

Maybe this can help:

import sys
import tempfile
from subprocess import Popen, PIPE


cmd = [sys.executable, '-c', 'print raw_input()']

# Using a temp file to give input data to the subprocess instead of stdin.write to avoid deadlocks.
with tempfile.TemporaryFile() as f:
    f.write('foobar')
    f.seek(0)  # Return at the start of the file so that the subprocess p1 can read what we wrote.
    p1 = Popen(cmd, stdin=f, stdout=PIPE)

p2 = Popen(cmd, stdin=p1.stdout, stdout=PIPE)
p3 = Popen(cmd, stdin=p2.stdout, stdout=PIPE)

# No order needed.
p1.stdout.close()
p2.stdout.close()

# Using communicate() instead of stdout.read to avoid deadlocks. 
print p3.communicate()[0]

Output:

$ python test.py
foobar

Hope this can be hepfull.