Python 3.4.3 subprocess.Popen get output of command without piping?

Question

I am trying to assign the output of a command to a variable without the command thinking that it is being piped. The reason for this is that the command in question gives unformatted text as output if it is being piped, but it gives color formatted text if it is being run from the terminal. I need to get this color formatted text.

So far I've tried a few things. I've tried Popen like so:

output = subprocess.Popen(command, stdout=subprocess.PIPE)
output = output.communicate()[0]
output = output.decode()
print(output)

This will let me print the output, but it gives me the unformatted output that I get when the command is piped. That makes sense, as I'm piping it here in the Python code. But I am curious if there is a way to assign the output of this command, directly to a variable, without the command running the piped version of itself.

I have also tried the following version that relies on check_output instead:

output = subprocess.check_output(command)
output = output.decode()
print(output)

And again I get the same unformatted output that the command returns when the command is piped.

Is there a way to get the formatted output, the output the command would normally give from the terminal, when it is not being piped?

Answer 1

A working polyglot example (works the same for Python 2 and Python 3), using pty.

import subprocess
import pty
import os
import sys

master, slave = pty.openpty()
# direct stderr also to the pty!
process = subprocess.Popen(
    ['ls', '-al', '--color=auto'],
    stdout=slave,
    stderr=subprocess.STDOUT
)

# close the slave descriptor! otherwise we will
# hang forever waiting for input
os.close(slave)

def reader(fd):
    try:
        while True:
            buffer = os.read(fd, 1024)
            if not buffer:
                return

            yield buffer

    # Unfortunately with a pty, an 
    # IOError will be thrown at EOF
    # On Python 2, OSError will be thrown instead.
    except (IOError, OSError) as e:
        pass

# read chunks (yields bytes)
for i in reader(master):
    # and write them to stdout file descriptor
    os.write(1, b'<chunk>' + i + b'</chunk>')

Answer 2

Yes, you can use the pty module.

>>> import subprocess
>>> p = subprocess.Popen(["ls", "--color=auto"], stdout=subprocess.PIPE)
>>> p.communicate()[0]
# Output does not appear in colour

With pty:

import subprocess
import pty
import os

master, slave = pty.openpty()
p = subprocess.Popen(["ls", "--color=auto"], stdout=slave)
p.communicate()
print(os.read(master, 100)) # Print 100 bytes
# Prints with colour formatting info

Note from the docs:

Because pseudo-terminal handling is highly platform dependent, there is code to do it only for Linux. (The Linux code is supposed to work on other platforms, but hasn’t been tested yet.)

A less than beautiful way of reading the whole output to the end in one go:

def num_bytes_readable(fd):
    import array
    import fcntl
    import termios
    buf = array.array('i', [0])
    if fcntl.ioctl(fd, termios.FIONREAD, buf, 1) == -1:
        raise Exception("We really should have had data")
    return buf[0]

print(os.read(master, num_bytes_readable(master)))

Edit: nicer way of getting the content at once thanks to @Antti Haapala:

os.close(slave)
f = os.fdopen(master)
print(f.read())

Edit: people are right to point out that this will deadlock if the process generates a large output, so @Antti Haapala's answer is better.

Answer 3

Using pexpect:

2.py:

import sys

if sys.stdout.isatty():
    print('hello')
else:
    print('goodbye')

subprocess:

import subprocess

p = subprocess.Popen(
    ['python3.4', '2.py'],
    stdout=subprocess.PIPE
)

print(p.stdout.read())

--output:--
goodbye

pexpect:

import pexpect

child = pexpect.spawn('python3.4 2.py')

child.expect(pexpect.EOF)
print(child.before)  #Print all the output before the expectation.

--output:--
hello

Here it is with grep --colour=auto:

import subprocess

p = subprocess.Popen(
    ['grep', '--colour=auto', 'hello', 'data.txt'],
    stdout=subprocess.PIPE
)

print(p.stdout.read())

import pexpect

child = pexpect.spawn('grep --colour=auto hello data.txt')
child.expect(pexpect.EOF)
print(child.before)

--output:--
b'hello world\n'
b'\x1b[01;31mhello\x1b[00m world\r\n'