Menu
I work in Python, and I want to find a workflow for enabling two processes (main-process and sub-process) to communicate with each other. By that, I mean the ability of main-process to send some data to sub-process (perhaps, by writing to sub-process's stdin) and the ability of sub-process to send some data back to the main one. This also implies that both can read the data sent to them (I was thinking of reading from stdin).
I was trying to use subprocess library, but it seems that it's intended to work with processes that are designed to give an output only once and then terminate, whereas I want to exchange data dynamically and shut the sub-process down only when such a command is received.
I've read lots of answers here on StackOverflow tackling problems closely related to mine, but none of them did I find satisfying, as the questions those answers were meant to were different from mine in one important detail: I need my main-process to be able to exchange data with its sub-process dynamically as many times as needed, not just once, which in turn implies that the sub-process should run until it receives a certain command from main-process to terminate.
I'm open to using third-party libraries, but it would be much better if you proposed a solution based solely on the Python Standard Library.
503
communicate() writes input (there is no input in this case so it just closes subprocess' stdin to indicate to the subprocess that there is no more input), reads all output, and waits for the subprocess to exit.
All sub processes are run in parallel. (To avoid this one has to wait explicitly for their completion.) They even can write into the log file at the same time, thus garbling the output. To avoid this you should let each process write into a different logfile and collect all outputs when all processes are finished.
wait() method is asynchronous, whereas subprocess. Popen. wait() method is implemented as a blocking busy loop; the universal_newlines parameter is not supported.
The subprocess module defines one class, Popen and a few wrapper functions that use that class. The constructor for Popen takes arguments to set up the new process so the parent can communicate with it via pipes. It provides all of the functionality of the other modules and functions it replaces, and more.
You want to make a Popen object with subprocess.PIPE for standard input and output and use its file objects to communicate—rather than using one of the cantrips like run (and the older, more specific ones like check_output). The challenge is avoiding deadlock: it’s easy to land in a situation where each process is trying to write, the pipe buffers fill (because no one is reading from them), and everything hangs. You also have to remember to flush in both processes, to avoid having a request or response stuck in a file object’s buffer.
Popen.communicate is provided to avoid these issues, but it supports only a single string (rather than an ongoing conversation). The traditional solution is select, but it also works to use separate threads to send requests and read results. (This is one of the reasons to use CPython threads in spite of the GIL: each exists to run while the other is blocked, so there’s very little contention.) Of course, synchronization is then an issue, and you may need to do some work to make the multithreaded client act like a simple, synchronous function call on the outside.
Note that both processes need to flush, but it’s enough if either implements such non-blocking I/O; one normally does that job in the process that starts the other because that’s where it’s known to be necessary (and such programs are the exception).
174
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
