How to get subprocess' stdout data asynchronously?

Question

I wrote a simple python script for my application and predefined some fast commands like make etc.

I've written a function for running system commands (linux):

def runCommand(commandLine):
    print('############## Running command: ' + commandLine)
    p = subprocess.Popen(commandLine, shell = True, stdout = subprocess.PIPE)
    print (p.stdout.read().decode('utf-8'))

Everything works well except a few things:

• I'm using cmake and it's output is colored. Any chances to save colors in output?

• I can look at output after process has finished. For example, make runs for a long period of time but I can see the output only after full compilation. How to do it asynchronously?

Answer 1

I'm not sure about colors, but here's how to poll the subprocess's stdout one line at a time:

import subprocess
proc = subprocess.Popen('cmake', shell=True, stdout=subprocess.PIPE)
while proc.poll() is None:
    output = proc.stdout.readline()
    print output

Don't forget to read from stderr as well, as I'm sure cmake will emit information there.

Answer 2

You're not getting color because cmake detects if its stdout is a terminal, if it's not it doesn't color its own output. Some programs give you an option to force coloring output. Unfortunately cmake does not, so you're out of luck there. Unless you want to patch cmake yourself.

Lots of programs do this, for example grep:

# grep test test.txt
test
 ^
 |
 |------- this word is red

Now pipe it to cat:

# grep test test.txt | cat
test
 ^
 |
 |------- no longer red

grep option --color=always to force color:

# grep test test.txt --color=always | cat
test
 ^
 |
 |------- red again