Python subprocess - write multiple stdin

Question

I need to open an R script and supply it with input formulated by a separate python script. The subprocess module seems to be a good way to do this.

I have encountered some puzzling results though, namely that I can apparently write once and only once via p.stdin. Here is what I have so far:

from subprocess import Popen, PIPE, STDOUT

p = Popen(['r --no-save'],stdin=PIPE,stdout=PIPE,stderr=PIPE,shell=True)
p.stdin.write("source('myrscript.R')\n")
p.stdin.write('myfirstinput')

What happens when I run this code is that the first instance of stdin.write() performs as expected (and opens my R script), but the second line does nothing, and the subprocess (really, the R script) exits with an error, indicating that the subprocessed received no input where input was expected and therefore terminated.

N.B. - In a perfect world, I would just interact directly through R, but this particular script requires complex inputs that cannot be entered directly for practical purposes. Also, rpy / rpy2 is not an option, because end-users of this script will not necessarily have access to that module or its dependencies. rscript is also not an option (for many reasons, but mainly because of variability in the end-users R configurations).

Finally, p.communicate is not an option, because apparently that will close the process after writing and I need to keep it open.

Thanks in advance

Answer 1

What you need is to call .communicate():

from subprocess import Popen, PIPE, STDOUT

p = Popen(
    ['r', '--nosave'],
    stdin=PIPE,
    stdout=PIPE,
    stderr=PIPE)
p.stdin.write("source('myrscript.R')\n")
p.stdin.write('myfirstinput\n')
p.stdin.write('q\n')

stdout, stderr = p.communicate()

print '---STDOUT---'
print stdout
print '---STDERR---'
print stderr
print '---'

Discussion

• I don't use the shell=True and it seems working with my fake R script since I don't have R install in my system. You might or might not need it.
• I prefer breaking the command line up into a list of string as shown, but a single string such as r --nosave will work as well; just don't do them both at the same time.
• Don't forget that stdin.write() does not write the new line character \n, you have to supply that yourself.

Update

My first attempt was off the mark, I hope this second attempt gets closer. As J.F. Sebastian suggested, you might want to use pexpect:

import pexpect
import sys

if __name__ == '__main__':
    prompt = '> ' # Don't know what the R prompt looks like
    lines = ['one', 'two', 'three']

    r = pexpect.spawn('r --no-save', logfile=sys.stdout)
    for line in lines:
        r.expect(prompt)
        r.sendline(line)

    # If you want to interact with your script, use these two lines
    # Otherwise, comment them out
    r.logfile = None # Turn off logging to sys.stdout
    r.interact()

Discussion

• You might need to install pexpect. I did it with pip install pexpect
• If you don't want to interact with the system, comment out the last two line, but make sure to send some signal for the R script to exit.
• spawn() returns a spawn object, see doc here.