Python: How to create and use a custom logger in python use logging module?

Question

I am trying to create a custom logger as in the code below. However, no matter what level I pass to the function, logger only prints warning messages. For example even if I set the argument level = logging.DEBUG by default my code fails to log the debug or info messages. Can someone point out the problem here.

import boto3
import logging


def get_logger(name=__name__, level=logging.DEBUG):
    # Create log handler
    logHandler = logging.StreamHandler()
    logHandler.setLevel(level)

    # Set handler format
    logFormat = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s", datefmt="%d-%b-%y")
    logHandler.setFormatter(logFormat)

    # Create logger
    logger = logging.getLogger(name)
    # Add handler to logger
    logger.addHandler(logHandler)

    # Stop propagating the log messages to root logger
    # logger.propagate = False

    return logger


def listBuckets():

    logThis = get_logger(level=logging.DEBUG)

    s3 = boto3.resource('s3')
    for bucket in s3.buckets.all():
        logThis.debug(msg='This message is from logger')
        print(bucket.name)


listBuckets()

Answer 1

You are missing the fact that a) every logger's ultimate ancestor is the root logger (which has level WARNING by default) and b) that both, loggers and handlers have levels.

The docs state:

When a logger is created, the level is set to NOTSET (which causes all messages to be processed when the logger is the root logger, or delegation to the parent when the logger is a non-root logger).

So, you create a logger and a StreamHandler with their default level NOTSET. Your logger is an implicit descendant of the root logger. You set the handler to level DEBUG, but not the logger using that handler. Since the level on your logger still is NOTSET, when a log event occurs, its chain of ancestors is traversed ...

... until either an ancestor with a level other than NOTSET is found, or the root is reached.

[...]

If the root is reached, and it has a level of NOTSET, then all messages will be processed. Otherwise, the root’s level will be used as the effective level.

Which means, you immediately end up at the root logger to determine the effective log level; it is set to WARNING as per the root logger's default. You can check this with the parent and level properties and the getEffectiveLevel method on the logger object:

logThis = get_logger()
print(logThis.parent)               # <RootLogger root (WARNING)>
print(logThis.level)                # 0 (= NOTSET)
print(logThis.getEffectiveLevel())  # 30 (= WARNING) from root logger

To have your logger handle the messages on and above the desired level itself, simply set it on the logger via logger.setLevel(level) in your get_logger function.