Persistent python subprocess

Question

Is there a way to make a subprocess call in python "persistent"? I'm calling a program that takes a while to load multiple times. So it would be great if I could just leave that program open and communicate with it without killing it.

The cartoon version of my python script looks like this:

for text in textcollection:
    myprocess = subprocess.Popen(["myexecutable"],
                stdin = subprocess.PIPE, stdout = subprocess.PIPE,
                stderr = None)
    myoutputtext, err = myprocess.communicate(input=text)

I need to process each text separately, so joining it all into one large text file and processing it once is not an option.

Preferably, if there's an option like this

myprocess = subprocess.Popen(["myexecutable"],
            stdin = subprocess.PIPE, stdout = subprocess.PIPE,
            stderr = None)    for text in textcollection:
for text in textcollection:
    myoutputtext, err = myprocess.communicate(input=text)

where I can leave the process open, I'd really appreciate it.

Answer 1

You can use myprocess.stdin.write() and myprocess.stdout.read() to communicate with your subprocess, you just need to be careful to make sure you handle buffering correctly to prevent your calls from blocking.

If the output from your subprocess is well-defined, you should be able to reliably communicate with it using line-buffering and myprocess.stdout.readline().

Here is an example:

>>> p = subprocess.Popen(['cat'], bufsize=1, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
>>> p.stdin.write('hello world\n')
>>> p.stdout.readline()
'hello world\n'
>>> p.stdout.readline()        # THIS CALL WILL BLOCK

An alternative to this method for Unix is to put the file handle in non-blocking mode, which will allow you to call functions like myprocess.stdout.read() and have it return data if any is available, or raise an IOError if there isn't any data:

>>> p = subprocess.Popen(['cat'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
>>> import fcntl, os
>>> fcntl.fcntl(p.stdout.fileno(), fcntl.F_SETFL, os.O_NONBLOCK)
0
>>> p.stdout.read()         # raises an exception instead of blocking
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IOError: [Errno 11] Resource temporarily unavailable

This would allow you to do something like this:

fcntl.fcntl(p.stdout.fileno(), fcntl.F_SETFL, os.O_NONBLOCK)
for text in textcollection:
    myprocess.stdin.write(text + '\n')
    while True:
        myoutputtext = ''
        try:
            myoutputtext += myprocess.stdout.read()
        except IOError:
            pass
        if validate_output(myoutputtext):
            break
        time.sleep(.1)    # short sleep before attempting another read

In this example, validate_output() is a function you would need to write that returns True if the data you have received so far is all of output that you expect to get.