How can I capture output and show it at the same time with Python?

Question

I have a pretty long running job, which runs for several minutes and then gets restarted. The task outputs various information which I capture like this:

output = subprocess.Popen(cmd,stdout=subprocess.PIPE).communicate()

The thing is, I will only get the entire output at a time. I would like to show output as the program is sending it to stdout, while still pushing it back in a buffer ( I need to check the output for the presence of some strings ). In Ruby I would do it like this:

IO.popen(cmd) do |io|
  io.each_line do |line|
    puts line
    buffer << line
  end
end

Answer 1

You can try something like this :

cmd = ["./my_program.sh"]
p = subprocess.Popen( cmd, shell=False, stdout=subprocess.PIPE)   # launch the process
while p.poll() is None:          # check if the process is still alive
    out = p.stdout.readline()    # if it is still alive, grab the output
    do_something_with(out)       # do what you want with it