Python - a subprocess writing to stdin so that the main program can read it from the stdin

Question

I have a python program which takes input from stdin. Now, I've to write another program and call it as a sub process so that every time I start this subprocess it should read data from another text file and write it to stdin and then my main program reads it from stdin and uses it.

out.py:

from subprocess import Popen, PIPE, STDOUT

import os

import sys

def read():
    p = Popen(['python', 'in.py'], stdout=PIPE, stdin=PIPE, stderr=STDOUT)

    out = p.communicate()

    print out[0]

def main():
    read()

if __name__ == "__main__":
    main()

in.py:

import sys,os

import subprocess

def main():

    f = open("ad.txt","r")

    for line in f:
        print line

if __name__ == "__main__":
    main()

Basically my problem here is the in.py reads the whole file and prints into STDOUT which I don't want and rather it should communicate one character once. If the ad.txt is like :

asdfgh

I should get "a" then "s" then "d" then "f" .. so on.. one by one character everytime I call the function read the next character from the file should be read. That's the thing! Phew!! Please help me and I got a heck of a work to do! Thanks in advance :D

Answer 1

The first program should read from a file (or stdin) and write to stdout (not stdin). Similarly, the second program should read from stdin and write to stdout. Now you can glue them together in the command line using the pipe symbol:

python first.py | python second.py

That's it! None of the programs has to be aware of the other; that's the beauty of using "pipelining".

Addendum: Shell pipelining works over a buffer. The shell listens to the first program and fills the buffer with its output. Simultaneously, the second program reads from the buffer as long as there is something to read, otherwise it waits idly. The communication is simultaneous and requires only a fixed size of memory.

Example code:

# first.py
import sys
for line in open("input.txt"):
    processed_line = process_line(line)
    sys.stdout.write(processed_line)

# second.py
import sys
for line in sys.stdin:
    processed_line = process_line(line)
    sys.stdout.write(processed_line)

These examples work over lines (which is better if you are working with text files). You could easily do the same byte by byte.