Using program variables in python logging configuration file

Question

I am trying to enable my python logging using the following:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import logging
import logging.config
import os
test_filename = 'my_log_file.txt'
try:
    logging.config.fileConfig('loggingpy.conf', disable_existing_loggers=False)
except Exception as e:
    # try to set up a default logger
    logging.error("No loggingpy.conf to parse", exc_info=e)
    logging.basicConfig(level=logging.WARNING, format="%(asctime)-15s %(message)s")
test1_log = logging.getLogger("test1")
test1_log.critical("test1_log crit")
test1_log.error("test1_log error")
test1_log.warning("test1_log warning")
test1_log.info("test1_log info")
test1_log.debug("test1_log debug")

I would like to use a loggingpy.conf file to control the logging like the following:

[loggers]
keys=root

[handlers]
keys=handRoot

[formatters]
keys=formRoot

[logger_root]
level=INFO
handlers=handRoot

[handler_handRoot]
class=FileHandler
level=INFO
formatter=formRoot
args=(test_filename,)

[formatter_formRoot]
format=%(asctime)s:%(name)s:%(process)d:%(lineno)d %(levelname)s %(message)s
datefmt=
class=logging.Formatter

Here I am trying to route the logging to the file named by the local "test_filename". When I run this, I get:

ERROR:root:No loggingpy.conf to parse
Traceback (most recent call last):
  File "logging_test.py", line 8, in <module>
    logging.config.fileConfig('loggingpy.conf', disable_existing_loggers=False)
  File "/usr/lib/python2.7/logging/config.py", line 85, in fileConfig
    handlers = _install_handlers(cp, formatters)
  File "/usr/lib/python2.7/logging/config.py", line 162, in _install_handlers
    args = eval(args, vars(logging))
  File "<string>", line 1, in <module>
NameError: name 'test_filename' is not defined
CRITICAL:test1:test1_log crit
ERROR:test1:test1_log error
WARNING:test1:test1_log warning

Reading the docs, it seems that the "args" value in the config is eval'd in the context of the logging package namespace rather than the context when fileConfig is called. Is there any decent way to try to get the logging to behave this way through a configuration file so I can configure a dynamic log filename (usually like "InputFile.log"), but still have the flexibility to use the logging config file to change it?

Answer 1

Even though it's an old question, I think this still has relevance. An alternative to the above mentioned solutions would be to use logging.config.dictConfig(...) and manipulating the dictionary.

MWE:

log_config.yml

version: 1
disable_existing_loggers: false
formatters:
  default:
    format: "%(asctime)s:%(name)s:%(process)d:%(lineno)d %(levelname)s %(message)s"
handlers:
  console:
    class: logging.StreamHandler
    formatter: default
    stream: ext://sys.stdout
    level: DEBUG
  file:
    class: logging.FileHandler
    formatter: default
    filename: "{path}/service.log"
    level: DEBUG
root:
  level: DEBUG
  handlers:
  - file
  - console

example.py

import logging.config
import sys
import yaml

log_output_path = sys.argv[1]

log_config = yaml.load(open("log_config.yml"))
log_config["handlers"]["file"]["filename"] = log_config["handlers"]["file"]["filename"].format(path = log_output_path)
logging.config.dictConfig(log_config)

logging.debug("test")

Executable as follows:

python example.py .

Result:

• service.log file in current working directory contains one line of log message.
• Console outputs one line of log message.

Both state something like this:
2016-06-06 20:56:56,450:root:12232:11 DEBUG test

Answer 2

The args statement is parsed using eval at logging.config.py _install_handlers. So you can add code into the args.

[handler_handRoot]
class=FileHandler
level=INFO
formatter=formRoot
args=(os.getenv("LOG_FILE","default_value"),)

Now you only need to populate the environment variable.

Answer 3

You could place the filename in the logging namespace with:

logging.test_filename = 'my_log_file.txt'

Then your existing loggingpy.conf file should work

You should be able to pollute the logging namespace with anything you like (within reason - i wouldn't try logging.config = 'something') in your module and that should make it referencable by the the config file.