Repeatedly write to STDIN and read STDOUT of a Subprocess without closing it

Question

I am trying to employ a Subprocess in Python for keeping an external script open in a Server-like fashion. The external script first loads a model. Once this is done, it accepts requests via STDIN and returns processed strings to STDOUT.

So far, I've tried

tokenizer = subprocess.Popen([tokenizer_path, '-l', lang_prefix], stdin=subprocess.PIPE, stdout=subprocess.PIPE)

However, I cannot use

tokenizer.stdin.write(input_string+'\n')
out = self._tokenizer.stdout.readline()

in order to repeatedly process input_strings by means of the subprocess – out will just be empty, no matter if I use stdout.read() or stdout.readline(). However, it works when I close the stdin with tokenizer.stdin.close() before reading STDOUT, but this closes the subprocess, which is not what I want as I would have to reload the whole external script again before sending another request.

Is there any way to use a subprocess in a server-like fashion in python without closing and re-opening it?

Answer 1

Thanks to this Answer, I found out that a slave handle must be used in order to properly communicate with the subprocess:

master, slave = pty.openpty()
tokenizer = subprocess.Popen(script, shell=True stdin=subprocess.PIPE, stdout=slave)
stdin_handle = process.stdin
stdout_handle = os.fdopen(master)

Now, I can communicate to the subprocess without closing it via

stdin_handle.write(input)
stdout_handle.readline() #gets the processed input

Answer 2

Your external script probably buffers its output, so you only can read it in the father when the buffer in the child is flushed (which the child must do itself). One way to make it flush its buffers is probably closing the input because then it terminates in a proper fashion and flushes its buffers in the process.

If you have control over the external program (i. e. if you can patch it), insert a flushing after the output is produced.

Otherwise programs sometimes can be made to not buffer their output by attaching them to a pseudo-TTY (many programs, including the stdlib, assume that when their output is going to a TTY, no buffering is wished). But this is a bit tricky.