Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to properly interact with a process using subprocess module

I'm having problems redirecting stdio of another program using subprocess module. Just reading from stdout results in hanging, and Popen.communicate() works but it closes pipes after reading/writing. What's the easiest way to implement this?

I was playing around with this on windows:

import subprocess
proc = subprocess.Popen('python -c "while True: print \'Hi %s!\' % raw_input()"',
                        shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE,
                        stderr=subprocess.STDOUT)
while True:
    proc.stdin.write('world\n')
    proc_read = proc.stdout.readline()
    if proc_read:
        print proc_read
like image 950
Ivan Baldin Avatar asked Jan 14 '09 14:01

Ivan Baldin


People also ask

What is sub process module?

The subprocess module allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes. This module intends to replace several older modules and functions: os. system os.

How do you call a subprocess in Python?

To start a new process, or in other words, a new subprocess in Python, you need to use the Popen function call. It is possible to pass two parameters in the function call. The first parameter is the program you want to start, and the second is the file argument.

What is communicate in subprocess?

communicate() writes input (there is no input in this case so it just closes subprocess' stdin to indicate to the subprocess that there is no more input), reads all output, and waits for the subprocess to exit.


1 Answers

Doesn't fit 100% to your example but helps to understand the underlying issue: Process P starts child C. Child C writes something to its stdout. stdout of C is a pipe which has a 4096 character buffer and the output is shorter than that. Now, C waits for some input. For C, everything is fine.

P waits for the output which will never come because the OS sees no reason to flush the output buffer of C (with so little data in it). Since P never gets the output of C, it will never write anything to C, so C hangs waiting for the input from P.

Fix: Use flush after every write to a pipe forcing the OS to send the data now.

In your case, adding proc.stdin.flush() in the main while loop and a sys.stdout.flush() in the child loop after the print should fix your problem.

You should also consider moving the code which reads from the other process into a thread. The idea here is that you can never know when the data will arrive and using a thread helps you to understand these issues while you write the code which processes the results.

At this place, I wanted to show you the new Python 2.6 documentation but it doesn't explain the flush issue, either :( Oh well ...

like image 59
Aaron Digulla Avatar answered Nov 13 '22 08:11

Aaron Digulla