How to call an external program in python and retrieve the output and return code?

Question

Look at the subprocess module: a simple example follows...

from subprocess import Popen, PIPE

process = Popen(["ls", "-la", "."], stdout=PIPE)
(output, err) = process.communicate()
exit_code = process.wait()

Following Ambroz Bizjak's previous comment, here is a solution that worked for me:

import shlex
from subprocess import Popen, PIPE

cmd = "..."
process = Popen(shlex.split(cmd), stdout=PIPE)
process.communicate()
exit_code = process.wait()

After some research, I have the following code which works very well for me. It basically prints both stdout and stderr in real time. Hope it helps someone else who needs it.

stdout_result = 1
stderr_result = 1


def stdout_thread(pipe):
    global stdout_result
    while True:
        out = pipe.stdout.read(1)
        stdout_result = pipe.poll()
        if out == '' and stdout_result is not None:
            break

        if out != '':
            sys.stdout.write(out)
            sys.stdout.flush()


def stderr_thread(pipe):
    global stderr_result
    while True:
        err = pipe.stderr.read(1)
        stderr_result = pipe.poll()
        if err == '' and stderr_result is not None:
            break

        if err != '':
            sys.stdout.write(err)
            sys.stdout.flush()


def exec_command(command, cwd=None):
    if cwd is not None:
        print '[' + ' '.join(command) + '] in ' + cwd
    else:
        print '[' + ' '.join(command) + ']'

    p = subprocess.Popen(
        command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, cwd=cwd
    )

    out_thread = threading.Thread(name='stdout_thread', target=stdout_thread, args=(p,))
    err_thread = threading.Thread(name='stderr_thread', target=stderr_thread, args=(p,))

    err_thread.start()
    out_thread.start()

    out_thread.join()
    err_thread.join()

    return stdout_result + stderr_result

Check out the subprocess module here: http://docs.python.org/library/subprocess.html#module-subprocess. It should get what you need done.

I've developed a little library (py-execute) that allows you to execute external programs, retrieve the output and the retcode and, at the same time get output in console in real time:

>>> from py_execute.process_executor import execute
>>> ret = execute('echo "Hello"')
Hello
>>> ret
(0, 'Hello\n')

You can avoid printing to console passing a mock user_io:

>>> from mock import Mock
>>> execute('echo "Hello"', ui=Mock())
(0, 'Hello\n')

I wrote it because with plain Popen (In Python 2.7) I was having trouble executing commands with a long output