Dynamic communication between main and subprocess in Python

Question

I work in Python, and I want to find a workflow for enabling two processes (main-process and sub-process) to communicate with each other. By that, I mean the ability of main-process to send some data to sub-process (perhaps, by writing to sub-process's stdin) and the ability of sub-process to send some data back to the main one. This also implies that both can read the data sent to them (I was thinking of reading from stdin).

I was trying to use subprocess library, but it seems that it's intended to work with processes that are designed to give an output only once and then terminate, whereas I want to exchange data dynamically and shut the sub-process down only when such a command is received.

I've read lots of answers here on StackOverflow tackling problems closely related to mine, but none of them did I find satisfying, as the questions those answers were meant to were different from mine in one important detail: I need my main-process to be able to exchange data with its sub-process dynamically as many times as needed, not just once, which in turn implies that the sub-process should run until it receives a certain command from main-process to terminate.

I'm open to using third-party libraries, but it would be much better if you proposed a solution based solely on the Python Standard Library.

Answer 1

You want to make a Popen object with subprocess.PIPE for standard input and output and use its file objects to communicate—rather than using one of the cantrips like run (and the older, more specific ones like check_output). The challenge is avoiding deadlock: it’s easy to land in a situation where each process is trying to write, the pipe buffers fill (because no one is reading from them), and everything hangs. You also have to remember to flush in both processes, to avoid having a request or response stuck in a file object’s buffer.

Popen.communicate is provided to avoid these issues, but it supports only a single string (rather than an ongoing conversation). The traditional solution is select, but it also works to use separate threads to send requests and read results. (This is one of the reasons to use CPython threads in spite of the GIL: each exists to run while the other is blocked, so there’s very little contention.) Of course, synchronization is then an issue, and you may need to do some work to make the multithreaded client act like a simple, synchronous function call on the outside.

Note that both processes need to flush, but it’s enough if either implements such non-blocking I/O; one normally does that job in the process that starts the other because that’s where it’s known to be necessary (and such programs are the exception).