Reading stderr of subprocess while it is executing

Question

I'd like to read what is written to stderr by a subprocess while it is executing.

However, when I use this script that I've written, stderr does not seem to have anything for me to read until the subprocess has exited.

#!/usr/bin/env python2

import sys
from subprocess import Popen, PIPE, STDOUT

if len(sys.argv) < 2:
    print "Please provide a command"
    sys.exit(1)

sub = Popen(sys.argv[1:], stdout=PIPE, stderr=STDOUT)

for i, line in enumerate(sub.stdout):
    sys.stdout.write("%d: %s" % (i, line))

edit:

Ok, I've gotten closer now. If I specify the number of bytes to read it overcomes the buffering.

#!/usr/bin/env python2

import sys 
from subprocess import Popen, PIPE, STDOUT

if len(sys.argv) < 2:
    print "Please provide a command"
    sys.exit(1)

sub = Popen(sys.argv[1:], stdout=PIPE, stderr=STDOUT)

i = 0 
while sub.poll() is None:
    line = sub.stdout.read(64)
    line.strip("\b")
    sys.stdout.write("%d: %s\n" % (i, line))
    i += 1

Snippet of output:

58:  86 q=21.0 size= 4541841kB time=00:00:22.08 bitrate=1685014.2kbi
frame=  567 fps= 86 q=22.0 size= 4543667kB time=00:00:2
frame=  621 fps= 87 q=20.0 sizs/s    
frame= 4545352kB time=00:00:26.11 bitrate=1425939.2kbits/s    
62: =  686 fps= 90 q=12.0 size= 4546970kB time=00:00:28.89 bitrate=1
frame=  758 fps= 93 q=25.0 size= 4548534kB t
frame=  794 fps= 92 bitrate=1168185.5kbits/s    
65:  q=27.0 size= 4550901kB time=00:00:33.40 bitrate=1115897.0kbits/
frame=  827 fps= 91 q=27.0 size= 4552324kB time=00:00:34.7
frame=  857 fps= 89 q=26.0 size=  
frame=  254kB time=00:00:36.12 bitrate=1032874.9kbits/s    
69: 892 fps= 88 q=25.0 size= 4556598kB time=00:00:37.36 bitrate=9988
frame=  948 fps= 89 q=19.0 size= 4558565kB time=
frame= 1006 fps= 90 q=19937320.4kbits/s    
72: .0 size= 4560139kB time=00:00:42.16 bitrate=885880.0kbits/s    
73: frame= 1060 fps= 91 q=19.0 size= 4561958kB time=00:00:44.49 bitr
frame= 1122 fps= 93 q=18.0 size= 4563460
frame= 1173 fps=0:47.08 bitrate=793898.4kbits/s

It looks like my problem now is that ffmpeg is using backspace characters or similar to mess with stdout. Not sure what's happening here.

Answer 1

I suggest to use sh module. It is very nice piece of software which wraps subprocesses in python and gives you pythonic, beautiful interface which you will love. Look at the docs.

If you really do not want sh module, use communicate method of Popen