Writing to same log file with Pyramid and rq

Question

I am using Pyramid and rq and I would like to log rq worker jobs in the same file as my Pyramid application.

My setup looks like this:

development.ini

# regular pyramid setup here...

[loggers]
keys = root, main

[handlers]
keys = console, debuglog

[formatters]
keys = generic

[logger_root]
level = DEBUG
handlers = console, debuglog

[logger_main]
level = DEBUG
handlers =
qualname = main

[handler_debuglog]
class = handlers.RotatingFileHandler
args = ('%(here)s/__logs/pyramid_debug.log','a')
level = DEBUG
formatter = generic

[formatter_generic]
format = %(asctime)s %(levelname)-8s [%(name)s][%(threadName)s] %(message)s

main.py

from bunch import Bunch
from redis import Redis
from rq import Queue, Worker
from helpers.rqworker import run_worker
from helpers.log import write_log

write_log('info', 'APP STARTED.')
q = Queue(connection=Redis())
req_obj = Bunch()
req_obj.remote_addr = self.request.remote_addr
job = q.enqueue(run_worker, req_obj, {'some_other': 'data'})
write_log('info', 'APP ENDED.')

helpers/rqworker.py

from helpers.log import write_log

def run_worker(req, data):
    write_log(req, 'debug', 'WORKER STARTED', separator=True)

helpers/log.py

import logging
log = logging.getLogger(__name__)


def write_log(request, loglevel, message=None, **kwargs):
    msg = '[{0}] {1}'.format(request.remote_addr, message)
    level = logging.getLevelName(loglevel.upper())

    if 'separator' in kwargs:
        log.log(level, '='*40)

    if message:
        log.log(level, msg)

    if 'stack' in kwargs:
        log.log(level, '{0} EXCEPTION {0}'.format('*'*15))
        log.log(level, kwargs['stack'])
        log.log(level, '{0} EXCEPTION END {0}'.format('*'*13))

When I run my app, I cannot see logs from rqworker.py (probably because it is a separate thread) so I'm wondering how can I (and is it possible to) achieve writing to same log file?

Answer 1

This is probably not useful to the OP anymore, but the way I've been able to redirect the output of a rq Worker to a custom file is by getting the logger like this:

logger = logging.getLogger('rq.worker')

You can add a new FileHandler to this logger that outputs on your file, and change the formatting too:

h = logging.FileHandler("/path/to/logfile.log")
formatter = logging.Formatter(...)
fh.setFormatter(formatter)
logger.addHandler(fh)

I haven't been able to remove the colouring of the output, though, which causes the log, when opened in a text editor, to be slightly cluttered with ANSI color codes.

Note that this removes the logging to the console. You can probably keep that too by adding another handler.