How can i create own logging instance for each thread?

Question

For example, i have these files with classes that i want to use (only example, maybe not working)

# helper_one.py

import logging

logger = logging.getLogger('HelperOne')

class HelperOne:
    def __init__(self, data):
        self.data = data

    def run(self):
        logger.debug('HelperOne::Run function with data: ' + self.data)
        return True

.

# controller_one.py

import logging

from helper_one import HelperOne

logger = logging.getLogger('ControllerOne')

class ControllerOne:
    def __init__(self, data):
        self.data = data

    def run(self):
        logger.debug('ControllerOne::Run function with data: ' + self.data)

        ho = HelperOne(self.data + '_data')
        return ho.run()

And i have a file which creates Threads

import logging 

from controller_one import ControllerOne

# this function creates threads (its just an example)
def run_task(task_id, data):
    logging.basicConfig(
        filename=os.path.join('logs', str(task_id) + '.log'),
        level=logging.DEBUG,
        format='%(asctime)s\t%(name)s\t%(levelname)s\t%(message)s'
    )

    result = ControllerOne(data)

    logging.debug('run_task: ' + result)

If i create logging instance like in my example, all log are written in a single file. How can I, for every thread, create its own logging instance with logging to its own file ?

Regards, Alex.

Answer 1

The logger name (the string passed to the getLogger function) can be anything you want. As the docs explain:

The name is potentially a period-separated hierarchical value, like foo.bar.baz (though it could also be just plain foo, for example). Loggers that are further down in the hierarchical list are children of loggers higher up in the list. For example, given a logger with a name of foo, loggers with names of foo.bar, foo.bar.baz, and foo.bam are all descendants of foo. The logger name hierarchy is analogous to the Python package hierarchy, and identical to it if you organise your loggers on a per-module basis using the recommended construction logging.getLogger(__name__). That’s because in a module, __name__ is the module’s name in the Python package namespace.

So, using __name__ as the name is recommended, but not required. And in fact you're already explicitly violating it:

logger = logging.getLogger('ControllerOne')

So, you could just use a separate name per thread, by putting a thread ID or name into the logger name. For example:

logger = logging.getLogger('ControllerOne.{}'.format(threading.get_ident())

Or, in your case, since you seem to have a unique task_id for each thread, you can just use that instead of the thread ID.

Of course this means you have to actually use the logger; you can't just call logging.debug. And you can't rely on the basicConfig; you're going to need to explicitly configure the logger in each thread. Since each thread creates its own independent ControllerOne instance, the obvious place to do that is in ControllerOne.__init__; in that function, you can call getLogger with a logger name that incorporates the thread or task ID, create and set a FileHandler that also incorporates the thread or task ID in the name, and store it as self.logger. Then, whenever you need to log something, you just do self.logger.debug(…).

If that all sounds like gobbledegook to you because you have no idea what a FileHandler is, you probably need to read at least the basic tutorial in the HOWTOs, but the advanced tutorial and cookbook are also very useful.