Python `tee` stdout of child process

Question

Is there a way in Python to do the equivalent of the UNIX command line tee? I'm doing a typical fork/exec pattern, and I'd like the stdout from the child to appear in both a log file and on the stdout of the parent simultaneously without requiring any buffering.

In this python code for instance, the stdout of the child ends up in the log file, but not in the stdout of the parent.

pid = os.fork()
logFile = open(path,"w")
if pid == 0:
  os.dup2(logFile.fileno(),1)  
  os.execv(cmd)

edit: I do not wish to use the subprocess module. I'm doing some complicated stuff with the child process that requires me call fork manually.

Answer 1

Here you have a working solution without using the subprocess module. Although, you could use it for the tee process while still using the exec* functions suite for your custom subprocess (just use stdin=subprocess.PIPE and then duplicate the descriptor to your stdout).

import os, time, sys

pr, pw = os.pipe()
pid = os.fork()

if pid == 0:
    os.close(pw)
    os.dup2(pr, sys.stdin.fileno())
    os.close(pr)
    os.execv('/usr/bin/tee', ['tee', 'log.txt'])
else:
    os.close(pr)
    os.dup2(pw, sys.stdout.fileno())
    os.close(pw)

    pid2 = os.fork()

    if pid2 == 0:
        # Replace with your custom process call
        os.execv('/usr/bin/yes', ['yes'])
    else:
        try:
            while True:
                time.sleep(1)
        except KeyboardInterrupt:
            pass

Note that the tee command, internally, does the same thing as Ben suggested in his answer: reading input and looping over output file descriptors while writing to them. It may be more efficient because of the optimized implementation and because it's written in C, but you have the overhead of the different pipes (don't know for sure which solution is more efficient, but in my opinion, reassigning a custom file-like object to stdout is a more elegant solution).

Some more resources:

• How do I duplicate sys.stdout to a log file in python?
• http://www.shallowsky.com/blog/programming/python-tee.html

Answer 2

In the following, SOMEPATH is the path to the child executable, in a format suitable for subprocess.Popen (see its docs).

import sys, subprocess


f = open('logfile.txt', 'w')
proc = subprocess.Popen(SOMEPATH, stdout=subprocess.PIPE)

while True:
    out = proc.stdout.read(1)
    if out == '' and proc.poll() != None:
        break
    if out != '':
        # CR workaround since chars are read one by one, and Windows interprets
        # both CR and LF as end of lines. Linux only has LF
        if out != '\r': f.write(out)
        sys.stdout.write(out)
        sys.stdout.flush()

Answer 3

Would an approach like this do what you want?

import sys

class Log(object):
    def __init__(self, filename, mode, buffering):
        self.filename = filename
        self.mode = mode
        self.handle = open(filename, mode, buffering)

    def write(self, thing):
        self.handle.write(thing)
        sys.stdout.write(thing)

You'd probably need to implement more of the file interface for this to be really useful (and I've left out properly defaulting mode and buffering, if you want it). You could then do all your writes in the child process to an instance of Log. Or, if you wanted to be really magic, and you're sure you implement enough of the file interface that things won't fall over and die, you could potentially assign sys.stdout to be an instance of this class. Then I think any means of writing to stdout, including print, will go via the log class.

Edit to add: Obviously if you assign to sys.stdout you will have to do something else in the write method to echo the output to stdout!! I think you could use sys.__stdout__ for that.

Answer 4

Oh, you. I had a decent answer all prettied-up before I saw the last line of your example: execv(). Well, poop. The original idea was replacing each child process' stdout with an instance of this blog post's tee class, and split the stream into the original stdout, and the log file:

http://www.shallowsky.com/blog/programming/python-tee.html

But, since you're using execv(), the child process' tee instance would just get clobbered, so that won't work.

Unfortunately for you, there is no "out of the box" solution to your problem that I can find. The closest thing would be to spawn the actual tee program in a subprocess; if you wanted to be more cross-platform, you could fork a simple Python substitute.

First thing to know when coding a tee substitute: tee really is a simple program. In all the true C implementations I've seen, it's not much more complicated than this:

while((character = read()) != EOF) {
    /* Write to all of the output streams in here, then write to stdout. */
}

Unfortunately, you can't just join two streams together. That would be really useful (so that the input of one stream would automatically be forwarded out of another), but we've no such luxury without coding it ourselves. So, Eli and I are going to have very similar answers. The difference is that, in my answer, the Python 'tee' is going to run in a separate process, via a pipe; that way, the parent thread is still useful!

(Remember: copy the blog post's tee class, too.)

import os, sys

# Open it for writing in binary mode.
logFile=open("bar", "bw")

# Verbose names, but I wanted to get the point across.
# These are file descriptors, i.e. integers.
parentSideOfPipe, childSideOfPipe = os.pipe()

# 'Tee' subprocess.
pid = os.fork()
if pid == 0:
    while True:
        char = os.read(parentSideOfPipe, 1)
        logFile.write(char)
        os.write(1, char)

# Actual command
pid = os.fork()
if pid == 0:
    os.dup2(childSideOfPipe, 1)
    os.execv(cmd)

I'm sorry if that's not what you wanted, but it's the best solution I can find.

Good luck with the rest of your project!