Python logging - filter log messages for all loggers

Question

I have a project where the underlying tools are logging, and I'm also logging (with different logger instances).

However, sometimes the logger I don't have access to exposes information which I'd like to strip out of the log (or replace with a placeholder).

Is there a way I can use a filter to do that for all python loggers in a project?

Here is my logging configuration in Django:

LOGGING_CONFIG = None
LOGGING = {
    "version": 1,
    "disable_existing_loggers": False,
    "formatters": {
        "my_formatter": {
            "format": "[%(asctime)s] %(message)s",
            "datefmt": "%d/%b/%Y %H:%M:%S",
        },
    },
    "handlers": {
        "console": {
            "level": "DEBUG",
            "class": "logging.StreamHandler",
            "formatter": "my_formatter",
        },
    },
    "loggers": {
        "my_logger": {
            "handlers": ["console"],
            "level": "DEBUG"
        },
    },
}

logging.config.dictConfig(LOGGING)

Really, my end goal is just to prevent certain things from popping up in logs by replacing them – if there's some other way to do that, feel free to share it.

Thanks!

Answer 1

If your main goal is to filter sensitive data, read Hiding Sensitive Data from Logs with Python. You can either implement a logging.Filter to prevent logging some records or implement a loggingFormatter to reduct only specific records' parts with a regex.

To apply the filters and formatters classes to all loggers define them in the dict config and add to all the handlers you have. Also, consider removing any undescribed handlers by setting disable_existing_loggers': True. See examples of custom formatters and filters in the Django logging docs.

Answer 2

Believe it or not, you do have access to those underlying projects' loggers! Because you are in django, Your settings can be loaded, and logging can be initialized, before those underlying projects' loggers are instantiated.

This is a two-step process. The first step is to identify the logger that is the source of the message you'd like to suppress. To do this, add name to my_formatter:

    "formatters": {
        "my_formatter": {
            "format": "[%(asctime)s] [%(name)s] %(message)s",
            "datefmt": "%d/%b/%Y %H:%M:%S",
        },
    },

Once we've found the name, we can define our filter.

import logging


class AwesomeFilter(logging.Filter):
    def filter(self, rec):
        if 'sensitive' in rec.msg:
            return 0
        # you may need to filter based on `getMessage()` if
        # you can't find the information in the pre-formatted msg field
        return 1

And now since you know the name of the logger that is producing the bad message, we can attach our AwesomeFilter to the logger:

LOGGING_CONFIG = None
LOGGING = {
    "version": 1,
    "disable_existing_loggers": False,
    "formatters": {
        "my_formatter": {
            "format": "[%(asctime)s] [%(name)s] %(message)s",
            "datefmt": "%d/%b/%Y %H:%M:%S",
        },
    },
    "handlers": {
        "console": {
            "level": "DEBUG",
            "class": "logging.StreamHandler",
            "formatter": "my_formatter",
        },
    },
    "loggers": {
        "my_logger": {
            "handlers": ["console"],
            "level": "DEBUG"
        },
        "name_of_logger_producing_bad_messages": {
            "filters": [ "app.filters.AwesomeFilter", ],
        },
    },
}

logging.config.dictConfig(LOGGING)

We use the trick of finding the name of the logger to control the output of logging for third-party libraries quite frequently. Good luck!