Run subprocess and print output to logging

Question

I am looking for the way to call shell scripts from python and write their stdout and stderr to file using logging. Here is my code:

import logging import tempfile import shlex import os  def run_shell_command(command_line):     command_line_args = shlex.split(command_line)      logging.info('Subprocess: \"' + command_line + '\"')      process_succeeded = True     try:         process_output_filename = tempfile.mktemp(suffix = 'subprocess_tmp_file_')         process_output = open(process_output_filename, 'w')          command_line_process = subprocess.Popen(command_line_args,\                                                 stdout = process_output,\                                                 stderr = process_output)         command_line_process.wait()         process_output.close()          process_output = open(process_output_filename, 'r')         log_subprocess_output(process_output)         process_output.close()          os.remove(process_output_filename)     except:         exception = sys.exc_info()[1]         logging.info('Exception occured: ' + str(exception))         process_succeeded = False      if process_succeeded:         logging.info('Subprocess finished')     else:         logging.info('Subprocess failed')      return process_succeeded 

And I am sure that there is the way to do it without creating temporary file to store process output. Any ideas?

Answer 1

You could try to pass the pipe directly without buffering the whole subprocess output in memory:

from subprocess import Popen, PIPE, STDOUT  process = Popen(command_line_args, stdout=PIPE, stderr=STDOUT) with process.stdout:     log_subprocess_output(process.stdout) exitcode = process.wait() # 0 means success 

where log_subprocess_output() could look like:

def log_subprocess_output(pipe):     for line in iter(pipe.readline, b''): # b'\n'-separated lines         logging.info('got line from subprocess: %r', line) 

Answer 2

I am sure that there is the way to do it without creating temporary file to store process output

You simply have to check for the documentation of Popen, in particular about stdout and stderr:

stdin, stdout and stderr specify the executed program’s standard input, standard output and standard error file handles, respectively. Valid values are PIPE, an existing file descriptor (a positive integer), an existing file object, and None. PIPE indicates that a new pipe to the child should be created. With the default settings of None, no redirection will occur; the child’s file handles will be inherited from the parent. Additionally, stderr can be STDOUT, which indicates that the stderr data from the child process should be captured into the same file handle as for stdout.

So you can see that you can either use a file object, or the PIPE value. This allows you to use the communicate() method to retrieve the output:

from StringIO import StringIO process = subprocess.Popen(arguments, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) output, error = process.communicate() log_subprocess_output(StringIO(output)) 

---

I'd rewrite your code as:

import shlex import logging import subprocess from StringIO import StringIO  def run_shell_command(command_line):     command_line_args = shlex.split(command_line)      logging.info('Subprocess: "' + command_line + '"')      try:         command_line_process = subprocess.Popen(             command_line_args,             stdout=subprocess.PIPE,             stderr=subprocess.STDOUT,         )          process_output, _ =  command_line_process.communicate()          # process_output is now a string, not a file,         # you may want to do:         # process_output = StringIO(process_output)         log_subprocess_output(process_output)     except (OSError, CalledProcessError) as exception:         logging.info('Exception occured: ' + str(exception))         logging.info('Subprocess failed')         return False     else:         # no exception was raised         logging.info('Subprocess finished')      return True