When Popen.communicate() is not enough?

Question

I found a lot of threads more or less related to this topic and still almost nothing like full answer... I'm looking for your advice on it.

So, here is my problem: I really need to communicate with a subprocess in both ways: I have to wright a lot of data to its input and get its output on the fly. There is no way around it. My subprocess is famous 'lame' mp3 encoder, the input is an hour or more long PCM sound generated by my function and the output is the mp3 file which also has to be sent to user chunk by chunk without waiting for encoder to finish.

According to docs, Popen.communicate() won't help me to handle IPC measured in dozens or hundreds of megabytes. At the same time, as I learned from here, this is very dangerous to try to create this approach from scratch since a lot of pitfalls are waiting: deadlocks, buffering, process management, etc.

So, my question is: is there some well known solution for this kind of problem: a python lib or a code example that really does solve the problem? May be at least there is an article or something clearly desribing most of problems one can run into trying to solve this by himself?

Thank you in advance, Ilya.

Answer 1

The easiest way is to split you own program in two: one which writes to LAME and the other which reads from lame and writes to the user. This is much easier to do than to have bidirectional communication.

If that doesn't work for you I have found development with named pipes much easier than traditional pipe IPC. It is easy to use various kind of plumbing during testing. Nonblocking I/O in Python 3 should make accessing them much easier.