Redirect stdout to a file only for a specific thread

Question

I have couple of processes with its own threads all writing to stdout and I am OK with it. Now, I have to add another thread, which is going to dump a lot of garbage to stdout and I don't want this.

Is there any way I can redirect the stdout for a single thread to a file?

Update

As mentioned by owobeid, I tried this... I am redirecting stderr to a file...

def startServer():
    fd = os.open("foo.txt", os.O_RDWR|os.O_CREAT )
    fd2 = 2
    os.dup2(fd, fd2)

    # rest of the code

if __name__ == '__main__':
    threading.Thread(target=startServer).start()
    raise Exception("My Exception")

Issue: stderr for the whole app is redirected. The Exception message also gets redirected into the file, even though it is outside the thread.

Answer 1

I stumbled across this post while searching for a way to do exactly this. I wanted to make an interactive Python console that uses ajax to proxy requests to a server, and return all output for only the thread executing. I ended up figuring it out and wanted to share my solution.

There's a class that comes with the werkzeug python library called local.LocalProxy that can make a module-level function behave like a property. For example, this will make sys.stdout behave exactly as normal, but it will be proxied through the LocalProxy class.

import sys
import werkzeug
sys.stdout = werkzeug.local.LocalProxy(lambda: sys.stdout)

Expanding on this, I then wrote a function in place of the lambda above to return a StringIO object if it's a different thread:

import threading
import sys
import cStringIO
import werkzeug

thread_proxies = {}
def redirect():
    ident = threading.currentThread().ident
    thread_proxies[ident] = cStringIO.StringIO()
    return thread_proxies[ident]

def proxy():
    ident = threading.currentThread().ident
    return thread_proxies.get(ident, sys.stdout)

sys.stdout = werkzeug.local.LocalProxy(proxy)

And then in any thread I want redirected, I can just call:

string_io = redirect()

And all of the output that would go to sys.stdout is instead now written to the StringIO object.

---

But wait! I need to capture sys.stdout, sys.__stdout__, sys.stderr, and sys.__stderr__, so I wrote this library, which I called stdout_helpers in my codebase:

import threading
import sys
import cStringIO
from werkzeug import local

# Save all of the objects for use later.
orig___stdout__ = sys.__stdout__
orig___stderr__ = sys.__stderr__
orig_stdout = sys.stdout
orig_stderr = sys.stderr
thread_proxies = {}


def redirect():
    """
    Enables the redirect for the current thread's output to a single cStringIO
    object and returns the object.

    :return: The StringIO object.
    :rtype: ``cStringIO.StringIO``
    """
    # Get the current thread's identity.
    ident = threading.currentThread().ident

    # Enable the redirect and return the cStringIO object.
    thread_proxies[ident] = cStringIO.StringIO()
    return thread_proxies[ident]


def stop_redirect():
    """
    Enables the redirect for the current thread's output to a single cStringIO
    object and returns the object.

    :return: The final string value.
    :rtype: ``str``
    """
    # Get the current thread's identity.
    ident = threading.currentThread().ident

    # Only act on proxied threads.
    if ident not in thread_proxies:
        return

    # Read the value, close/remove the buffer, and return the value.
    retval = thread_proxies[ident].getvalue()
    thread_proxies[ident].close()
    del thread_proxies[ident]
    return retval


def _get_stream(original):
    """
    Returns the inner function for use in the LocalProxy object.

    :param original: The stream to be returned if thread is not proxied.
    :type original: ``file``
    :return: The inner function for use in the LocalProxy object.
    :rtype: ``function``
    """
    def proxy():
        """
        Returns the original stream if the current thread is not proxied,
        otherwise we return the proxied item.

        :return: The stream object for the current thread.
        :rtype: ``file``
        """
        # Get the current thread's identity.
        ident = threading.currentThread().ident

        # Return the proxy, otherwise return the original.
        return thread_proxies.get(ident, original)

    # Return the inner function.
    return proxy


def enable_proxy():
    """
    Overwrites __stdout__, __stderr__, stdout, and stderr with the proxied
    objects.
    """
    sys.__stdout__ = local.LocalProxy(_get_stream(sys.__stdout__))
    sys.__stderr__ = local.LocalProxy(_get_stream(sys.__stderr__))
    sys.stdout = local.LocalProxy(_get_stream(sys.stdout))
    sys.stderr = local.LocalProxy(_get_stream(sys.stderr))


def disable_proxy():
    """
    Overwrites __stdout__, __stderr__, stdout, and stderr with the original
    objects.
    """
    sys.__stdout__ = orig___stdout__
    sys.__stderr__ = orig___stderr__
    sys.stdout = orig_stdout
    sys.stderr = orig_stderr

And now at the start of my app I call:

stdout_helpers.enable_proxy()

And in any thread I now call:

string_io = stdout_helpers.redirect()

Answer 2

This answer allow redirecting to specific file, but also to have multiple outputs (sys.stdout + file), and also to redirect main thread logs to each thread filelog (which was very useful in my case).

First of all, let's define a class for new redirection :

class SysRedirect(object):
    def __init__(self):
        self.terminal = sys.stdout                  # To continue writing to terminal
        self.log={}                                 # A dictionary of file pointers for file logging

    def register(self,filename):                    # To start redirecting to filename
        ident = threading.currentThread().ident     # Get thread ident (thanks @michscoots)
        if ident in self.log:                       # If already in dictionary :
            self.log[ident].close()                 # Closing current file pointer
        self.log[ident] = open(filename, "a")       # Creating a new file pointed associated with thread id

    def write(self, message):
        self.terminal.write(message)                # Write in terminal (comment this line to remove terminal logging)
        ident = threading.currentThread().ident     # Get Thread id
        if ident in self.log:                       # Check if file pointer exists
            self.log[ident].write(message)          # write in file
        else:                                       # if no file pointer 
            for ident in self.log:                  # write in all thread (this can be replaced by a Write in terminal)
                 self.log[ident].write(message)  
     def flush(self):
            #this flush method is needed for python 3 compatibility.
            #this handles the flush command by doing nothing.
            #you might want to specify some extra behavior here.
            pass    

Then, I only need to initialise in my main thread

sys.stdout=SysRedirect()

Then in each thread, I only have to register and specifying a filename

sys.stdout.register('threadX.log')

And in main Thread, I can redirect for example to :

sys.stdout.register('mainthread.log')

But in my case I prefer not to register the main Thread so all std in main thread is written in all others logs