Get output colored text with Popen

Question

I'm using popen to make a plugin, this plugin used a external program who shows some colored text in the output.

The output is something like this:

avr-g++ -o .pioenvs\uno\FrameworkArduino\HardwareSerial.o -c -std=gnu++11 -fno-exceptions -fno-threadsafe-statics -g -Os -Wall -ffunction-sections -fdata-sections -mmcu=atmega328p -DF_CPU=16000000L -DARDUINO_ARCH_AVR -DARDUINO_AVR_UNO -DARDUINO=10607 -I.pioenvs\uno\FrameworkArduino -I.pioenvs\uno\FrameworkArduinoVariant .pioenvs\uno\FrameworkArduino\HardwareSerial.cpp    
avr-g++ -o .pioenvs\uno\FrameworkArduino\HardwareSerial0.o -c -std=gnu++11 -fno-exceptions -fno-threadsafe-statics -g -Os -Wall -ffunction-sections -fdata-sections -mmcu=atmega328p -DF_CPU=16000000L -DARDUINO_ARCH_AVR -DARDUINO_AVR_UNO -DARDUINO=10607 -I.pioenvs\uno\FrameworkArduino -I.pioenvs\uno\FrameworkArduinoVariant .pioenvs\uno\FrameworkArduino\HardwareSerial0.cpp
=============================================
Path\file.cpp: in function 'void loop()':
Path\file.cpp:23:2 error: expected ';' before '}' token
}
^
=============================================

All inside "=" is in red and yellow.

When I run the command in the command console, I can see the full output, but when I use Popen I only can get the uncolored text

This is how I'm using Popen

process = subprocess.Popen(command, stdin=subprocess.PIPE, stdout=subprocess.PIPE, cwd=self.cwd, universal_newlines=True, shell=True)
output = process.communicate()

stdout = output[0]
stderr = output[1]

print(stdout)
print(stderr)

I would like to get the text even if it's not colored, the important is to get the full log.

Any suggestion will be appreciated

Answer 1

You are not able to get all that message because a part of your command output is not regular output, it is considered as Error (or log or debug messages)

Now you can either add stderr=subprocess.PIPE to your Popen's parameters, which will put all the errors in your stderr variable:

process = subprocess.Popen(command, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, cwd=self.cwd, universal_newlines=True, shell=True)
output = process.communicate()

stdout = output[0]
stderr = output[1]

print(stdout)
print(stderr)

OR if you want to have all the errors and output just like what you see in the console, add 2>&1 at the end of your command. something like:

avr-g++ -o .pioenvs\uno\FrameworkArduino\HardwareSerial.o -c -std=gnu++11 -fno-exceptions -fno-threadsafe-statics -g -Os -Wall -ffunction-sections -fdata-sections -mmcu=atmega328p -DF_CPU=16000000L -DARDUINO_ARCH_AVR -DARDUINO_AVR_UNO -DARDUINO=10607 -I.pioenvs\uno\FrameworkArduino -I.pioenvs\uno\FrameworkArduinoVariant .pioenvs\uno\FrameworkArduino\HardwareSerial.cpp 2>&1

Answer 2

When I run the command in the command console, I can see the full output, but when I use Popen I only can get the uncolored text

There could be two issues:

1. the colored text happens to be produced on stderr, not stdout. To capture stderr, set stderr=PIPE, as @Rahman suggested

2. avr-g++ command may disable colors if it detects that the output is not a tty e.g., if it is redirected to a pipe like in your case. To enable colors, pass the command-line option -fdiagnostics-color=always as @PM 2Ring suggested.

   Or (less portable but it may work with more commands) provide a pseudo-tty to hoodwink the avr-g++ command into thinking that it runs interactively (and therefore it should enable colors). You could do it using pty.spawn(), pexpect module:

   import pexpect # $ pip install pexpect
   
   output, exitstatus = pexpect.runu(command, withexitstatus=1)
   

   or (more low-level) using pty.openpty() + subprocess module.