Configuring the logging of a third party script

Question

I have a third party python console script, which source I don't want to modify.

But I want to configure the logging which is done by the script and its libraries. The script uses the standard python logging, but does not support configuration of it.

The script uses this pattern:

import logging
logger=logging.getLogger(__name__)

Use cases:

• I want INFO messages of file foo.py to be ignored.
• I want to include the PID in the loggings messages.

How can I configure the logging, if I don't want to modify the sources of the console script?

The script gets called via cron.

How can I configure the logging if this script?

Important

Creating a wrapper script like in this answer is not a solution for me.

The linux process hierarchy looks like this:

Cron -> third_party_script

There should be any "glue", "wrapping" or "dirty-hack" script between cron and third_party_script.

Why obtrusive/netpicking?

I want to practice "separation of concerns". I want to be able to configure logging one time and in one place. This configuration should get used by all python code of a virtualenv. Writing a wrapper would be a work-around. I want a solution.

Update

Several months later I think a pth file would an simple answer.

Answer 1

A library isn't supposed to configure logging - that's up to the application developer. Inbar Rose's answer isn't quite right. If the module you're referring to is called foo, then the reference to __name__ in its getLogger call will be passing in foo. So in your configuration code, you would need to do the equivalent of

logging.getLogger('foo').setLevel(logging.WARNING)

To include the PID in the logs, just ensure that you use an appropriate format string for your Formatters, i.e. one which includes %(process)d. A simple example would be:

logging.basicConfig(format='%(process)d %(message)s')

Note that you can't write to the same log file from multiple processes concurrently - you may need to consider an alternative approach if you want to do this.

Update: An application developer is someone who writes Python code which is not the library, but is invoked by e.g. a user or another script via a command line or other means of creating a Python process.

To use the code I posted above, there is no need to wrap or modify the third-party code, as long as it's a library. For example, in the main script which invokes the third-party library:

if __name__ == '__main__':
    # configure logging here
    # sets the third party's logger to do WARNING or greater
    # replace 'foo' with whatever the top-level package name your
    # third party package uses
    logging.getLogger('foo').setLevel(logging.WARNING)
    # set any other loggers to use INFO or greater,
    # unless otherwise configured explicitly
    logging.basicConfig(level=logging.INFO, format='%(process)d %(message)s')
    # now call the main function (or else inline code here)
    main()

If the third party code runs via cron, it's not library code - it's an application, and you are probably out of luck.

Answer 2

I asked this question several months ago. Unfortunately I got no answer which satisfied me.

The distinction between using logging and setting it up is important for me.

This is my solution: In our context we set up logging in a method which gets called in usercustomize.py.

This way the optional plugins can use the logging without the need to set it up.

This almost solved all my needs.

Up to now I found no better way than usercustomize.py. My perfect solution would be something I would call virtualenvcustomize.py: Some initialization code which gets run if the interpreter loads virtualenv. Up to now I could not find such a hook. Please let me know if you have a solution.

Answer 3

Several possibilities:

Wrapper

If you can edit your cron table you could create a small script in python that get the lib logger, remove the existing log handler and hook your custom handler on it:

# Assumes the lib defines a logger object
from third_party_lib import *

# Note this assumes only one handler was defined by the lib
logger.removeHandler(logger.handlers[0])

# Then we can hook our custom format handler
custom_handler = logging.StreamHandler(sys.stdout)
custom_handler.setFormatter(logging.Formatter(format = '%(asctime)s %(levelname)s %(name)s %(process)d: %(message)s', None))
logger.addHandler(custom_handler)
logger.setLevel(logging.WARNING)

Also keep in mind this suppose the lib does not re-declare the logger on the way.

Dynamic code edit

If you do not have the possibility to modify the cron call, you might be able to do dynamic code edit but that is equivalent to editing the file by hand (hacky):

• get the third party file containing the logger configuration
• modify that and save back the modified version
• the cron job starts the task using third party code
• after execution of the cron job, you restore the file to their original state.