Storing logger messages in a string

Question

I wanted to store all the intermediate log messages (warn, info, error) to a string in Python, and report those log messages to the console at the end of program.

I tried to follow the steps outlined in http://opensourcehacker.com/2011/02/23/temporarily-capturing-python-logging-output-to-a-string-buffer/ but was unsuccessful .

Could somebody tell me a short, clean way to do this?

This is what I've tried for now:

log = logging.getLogger('basic_logger')
log.setLevel(logging.DEBUG)
report = ""

memory_handler = logging.handlers.MemoryHandler(1024*20, logging.ERROR, report)
memory_handler.setLevel(logging.DEBUG)
log.addHandler(memory_handler)
    
log.info("hello world")
    
memory_handler.flush()
    
print "report:", report

Answer 1

It can be as simple as logging to a StringIO object:

import logging
try:
    from cStringIO import StringIO      # Python 2
except ImportError:
    from io import StringIO

log_stream = StringIO()    
logging.basicConfig(stream=log_stream, level=logging.INFO)

logging.info('hello world')
logging.warning('be careful!')
logging.debug("you won't see this")
logging.error('you will see this')
logging.critical('critical is logged too!')

print(log_stream.getvalue())

Output

INFO:root:hello world
WARNING:root:be careful!
ERROR:root:you will see this
CRITICAL:root:critical is logged too!

---

If you want to log only those messages at levels WARN, INFO and ERROR you can do it with a filter. LevelFilter below checks each log record's level no, allowing only those records of the desired level(s):

import logging
try:
    from cStringIO import StringIO      # Python 2
except ImportError:
    from io import StringIO

class LevelFilter(logging.Filter):
    def __init__(self, levels):
        self.levels = levels

    def filter(self, record):
        return record.levelno in self.levels

log_stream = StringIO()    
logging.basicConfig(stream=log_stream, level=logging.NOTSET)
logging.getLogger().addFilter(LevelFilter((logging.INFO, logging.WARNING, logging.ERROR)))

logging.info('hello world')
logging.warning('be careful!')
logging.debug("you won't see this")
logging.error('you will see this')
logging.critical('critical is no longer logged!')

print(log_stream.getvalue())

Output

INFO:root:hello world
WARNING:root:be careful!
ERROR:root:you will see this

Answer 2

Note that solutions involving basicConfig set attributes of the root logger which all other loggers inherit from, this can be unwanted because libraries will also log to it. My use case is a website that calls a data processing module, and I only want to capture that module's logs specifically. This also has the advantage of allowing existing handlers that log to file and the terminal to persist:

import io, logging
from django.http import HttpResponse

log_stream = io.StringIO()
log_handler = logging.StreamHandler(log_stream)
logging.getLogger('algorithm.user_output').addHandler(log_handler)

algorithm()
return HttpResponse(f'<pre>{log_stream.getvalue()}</pre>')

In algorithm.py:

logger = logging.getLogger(__name__ + '.user_output')  # 'algorithm.user_output'

Answer 3

You can also write your own stream class. As https://docs.python.org/2/library/logging.handlers.html says, only writeand flushare used for the streaming.

Example:

import logging

class LogStream(object):
    def __init__(self):
        self.logs = ''

    def write(self, str):
        self.logs += str

    def flush(self):
        pass

    def __str__(self):
        return self.logs

log_stream = LogStream()
logging.basicConfig(stream=log_stream, level=logging.DEBUG)

log = logging.getLogger('test')
log.debug('debugging something')
log.info('informing user')

print(log_stream)

Outputs:

DEBUG:test:debugging something
INFO:test:informing user