Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

why does python.subprocess hang after proc.communicate()?

I've got an interactive program called my_own_exe. First, it prints out alive, then you input S\n and then it prints out alive again. Finally you input L\n. It does some processing and exits.

However, when I call it from the following python script, the program seemed to hang after printing out the first 'alive'.

Can anyone here tell me why this is happening?

// after reading the follow ups (thank you guys), i modified the code as following:

import subprocess
import time

base_command = "./AO_FelixStrategy_UnitTest --bats 31441 --chix 12467 --enxutp 31884 --turq 26372 --symbol SOGN --target_date " + '2009-Oct-16'
print base_command

proc2 = subprocess.Popen(base_command, shell=True , stdin=subprocess.PIPE,)

time.sleep(2);
print "aliv"
proc2.communicate('S\n')

print "alive"
time.sleep(6)

print "alive"
print proc2.communicate('L\n')
time.sleep(6)

the program now goes well with the first input 'S\n', but then stopped, and I the second 'L\n' is kinda ignored.

Can anyone give me an idea why it's like this?

like image 730
James Bond Avatar asked Mar 09 '10 11:03

James Bond


People also ask

Does subprocess wait for command to finish?

subprocess. run() is synchronous which means that the system will wait till it finishes before moving on to the next command.

Do you need to close Popen?

Popen do we need to close the connection or subprocess automatically closes the connection? Usually, the examples in the official documentation are complete. There the connection is not closed. So you do not need to close most probably.

What is difference between subprocess Popen and call?

Popen is more general than subprocess. call . Popen doesn't block, allowing you to interact with the process while it's running, or continue with other things in your Python program. The call to Popen returns a Popen object.

What is communicate in subprocess?

communicate() writes input (there is no input in this case so it just closes subprocess' stdin to indicate to the subprocess that there is no more input), reads all output, and waits for the subprocess to exit.


1 Answers

From the docs for communicate:

Interact with process: Send data to stdin. Read data from stdout and stderr, until end-of-file is reached. Wait for process to terminate.

So after communicate() runs, the process has been terminated.

If you want to write and read without waiting for the process to stop:

  • Don't ever use shell=True - it needlessy invokes a shell to in turn call your program, so there will be another process between you and your program. That has lots of unpleasant side-effects. The default is shell=False so you should stick with that. Change your Popen line to:

    p = subprocess.Popen(["./AO_FelixStrategy_UnitTest",
                          "--bats", "31441", "--chix", "12467",
                          "--enxutp", "31884", "--turq", "26372",
                          "--symbol", "SOGN", "--target_date", '2009-Oct-16'],
                         stdin=subprocess.PIPE, 
                         stdout=subprocess.PIPE)
    
  • Use p.stdin.write to write to the process. Use p.stdout.read to read from it.

  • Calling p.stdout.read if there's nothing to read will block. Calling p.stdin.write if the write buffer is full will block. So you have to make sure you have something to read/write - you do that on unix OS by using select. On windows you unfortunately must resort to threads. At least that is what Popen.communicate does internally.
  • If you didn't write AO_FelixStrategy_UnitTest then you have possible additional problems:
    • It could be reading from somewhere else, not standard input. Some programs read directly from the terminal, others use some OS API to read. That means data written to stdin won't go to the program. This is often true for password prompts.
    • Remember that you have to account for AO_FelixStrategy_UnitTest buffers. By default standard C PIPE communication is buffered so you may not see any output until after you've closed the input side (by doing p.stdin.close(). Unless AO_FelixStrategy_UnitTest flushes the output periodically.

Here's some example code, based on what you describe. It could work depending on how AO_FelixStrategy_UnitTest was developed:

p = subprocess.Popen(["./AO_FelixStrategy_UnitTest",
                      "--bats", "31441", "--chix", "12467",
                      "--enxutp", "31884", "--turq", "26372",
                      "--symbol", "SOGN", "--target_date", '2009-Oct-16'],
                     stdin=subprocess.PIPE, 
                     stdout=subprocess.PIPE)
output = p.communicate('S\nL\n')[0]
print output
like image 133
nosklo Avatar answered Oct 08 '22 19:10

nosklo