How can I temporarily redirect the output of logging in Python?

Question

There's already a question that answers how to do this regarding sys.stdout and sys.stderr here: https://stackoverflow.com/a/14197079/198348

But that doesn't work everywhere. The logging module seems to output to sys.stdout and sys.stderr, but I can't capture it with the context manager above.

In the following example code, I'm trying to capture all output inside the context manager, failing to do so for the logger statements:

from __future__ import print_function
import contextlib
import sys
import logging
from StringIO import StringIO

# taken from https://stackoverflow.com/a/14197079/198348
@contextlib.contextmanager
def stdout_redirect(where):
    prev_stdout = sys.stdout
    prev_stderr = sys.stderr
    prev_stdout.flush()
    sys.stdout = where
    sys.stderr = where
    try:
        yield where
    finally:
        where.flush()
        sys.stdout = prev_stdout
        sys.stderr = prev_stderr

logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger()

print("\t\tOUTSIDE: stdout", file=sys.stdout)
print("\t\tOUTSIDE: stderr", file=sys.stderr)
logger.info("\tOUTSIDE: info")
logger.debug("\tOUTSIDE: debug")
logger.warn("\tOUTSIDE: warn")
logger.error("\tOUTSIDE: error")
logger.critical("\tOUTSIDE: critical")

print("=============== DIVIDER ================")

s = ""
with stdout_redirect(StringIO()) as new_stdout:
    print("\t\tINSIDE: stdout", file=sys.stdout)
    print("\t\tINSIDE: stderr", file=sys.stderr)
    logger.info("\tINSIDE: info")
    logger.debug("\tINSIDE: debug")
    logger.warn("\tINSIDE: warn")
    logger.error("\tINSIDE: error")
    logger.critical("\tINSIDE: critical")


print("=============== DIVIDER ===============")
print(new_stdout.getvalue())

print("=============== LOGGING ===============")

print(logger.handlers)
print(logger.root.handlers)

How can I temporarily redirect the output of the logger(s) that spit out to stdout and capture them? I took a look at logging/init.py, but it doesn't immediately tell me what I need to do.

My motivation for doing this is that I want to equip a crufty big codebase with tests, each of which captures the spurious amounts of logging output that each test invokes. I can capture external programs, but I can't seem to capture the tests that I run inside nose.

Rewriting the verbose parts isn't an option right now, but is definitely a goal for further down the road.

Edit, regarding ubuntu

Here's what I've tried running with nosetests:

from __future__ import print_function
import sys

def test_funky_shurane():
    import logging
    logging.basicConfig(level=logging.DEBUG)
    logging.info("===== shurane info")
    logging.warn("===== shurane warn")
    logging.error("===== shurane error")
    logging.critical("===== shurane critical")
    print("===== shurane stdout", file=sys.stdout)
    print("===== shurane stderr", file=sys.stderr)
    assert True

And then running the above with:

nosetests test_logging.py
nosetests --nocapture test_logging.py

Answer 1

the logging.basicConfig() is a convenience that sets up some logger handling in a very simple way. If you need a little more than that, you shouldn't use basicConfig(). That's not a big deal, because it doesn't do a whole lot. What we need is to configure logging for both streams;

import logging, sys
fmt = logging.Formatter(BASIC_FORMAT)

hdlr_stderr = logging.StreamHandler(sys.stderr)
hdlr_stderr.setFormatter(fmt)
hdlr_stdout = logging.StreamHandler(sys.stdout)
hdlr_stdout.setFormatter(fmt)
root.addHandler(hdlr_stderr)
root.addHandler(hdlr_stdout)
root.setLevel(logging.DEBUG)

By default, loggers log all messages that they receive; but initially, we don't want to log any messages to sys.stdout:

hdlr_stdout.level = float('inf')  # larger than any log level; nothing gets logged

Then, your context manager might look a bit like:

@contextlib.contextmanager
def redirect_stderr_logging(where):
    hdlr_stderr.level = float('inf')
    hdlr_stdout.level = logging.NOTSET
    try:
        yield where
    finally:
        hdlr_stderr.level = logging.NOTSET
        hdlr_stdout.level = float('inf')