Can I use the generic logging.Logger() in Celery tasks?

Question

the doc says

A special logger is available named “celery.task”, you can inherit from this logger to automatically get the task name and unique id as part of the logs.

This is hardly enough. Is there a more detailed info available? Specifically, what handlers and format strings it defines by default? Why would I want to inherit from it? Can I use a generic logging.Logger() instead? What are the best practices for logging to a file from celery tasks (not Django)? etc. Thank you.

Answer 1

Can I use a generic logging.Logger() instead?

Yes, you can use the generic Python/Django logger. Instead of logger = get_task_logger(__name__) just choose the getLogger:

import logging
logger = logging.getLogger(__name__)

---

Why would I want to inherit from it?

The advantage of using celery.task is:

automatically get the task name and unique id as part of the logs.

which itself uses the standard Python logging lib.

---

This is hardly enough. Is there a more detailed info available? Specifically, what handlers and format strings it defines by default?

By default it's using the WatchedFileHandler from logging.handlers You can check in the celery.app.log documentation, it's getting set in the setup_handler method.

---

What are the best practices for logging to a file from celery tasks (not Django)? etc

1. Directly in the module - You can simply put everything at the top of your module - define a handler, and while getting the logger assign it:

import logging

# ---- set up logging to file ---
logging.basicConfig(level=logging.DEBUG,
                    format='%(asctime)s %(name)-12s %(levelname)-8s %(message)s',
                    datefmt='%m-%d %H:%M',
                    filename='/temp/myapp.log',
                    filemode='w')
# --- define a Handler ---
console = logging.StreamHandler()
console.setLevel(logging.INFO)
# --- set a format which is simpler for console use ---
formatter = logging.Formatter('%(name)-12s: %(levelname)-8s %(message)s')
# --- tell the handler to use this format ---
console.setFormatter(formatter)
# --- add the handler to the logger ---
logging.getLogger('').addHandler(console)

2. Use logging_config dictionary - dictConfig: a much more good looking practice.

import logging
from logging.config import dictConfig

logging_config = dict(
    version = 1,
    formatters = {
        'f': {'format':
              '%(asctime)s %(name)-12s %(levelname)-8s %(message)s'}
        },
    handlers = {
        'h': {'class': 'logging.StreamHandler',
              'formatter': 'f',
              'level': logging.DEBUG}
        },
    root = {
        'handlers': ['h'],
        'level': logging.DEBUG,
        },
)

dictConfig(logging_config)

logger = logging.getLogger()
logger.debug('often makes a very good meal of %s', 'visiting tourists')

3. Using a separate logging format file - *.ini:

[loggers]
# list of loggers
keys=root,log02

[handlers]
# list of handlers    
keys=hand01,hand02

[formatters]
# list of formatters    
keys=form01,form02

[logger_root]
# config for 'root' logger    
level=NOTSET
handlers=hand01

[handler_hand01]
# config for handler hand01    
class=StreamHandler
level=NOTSET
formatter=form01
args=(sys.stdout,)

[handler_hand02]
## config for handler hand02     
class=FileHandler
level=DEBUG
formatter=form02
args=('python.log', 'w')

[formatter_form01]
# config for formatter form01    
format=F1 %(asctime)s %(levelname)s %(message)s
datefmt=
class=logging.Formatter

4. A separate class - Last but not least, a more elegant solution is to have all this as separate module (Class), and just import it with specific flags where ever required:

import logging
from config import LOG, LOGGING

class Logger:
    def __init__(self, logf=None, logger_name=None, debug=None, rotation=None):
        self.logfile = logf
        self.logger_name = logger_name if logger_name else logf
        self.logger = self.get_logger(rotation=rotation) if rotation else self.get_logger()
        self.debug = debug if debug else False
        self.debug = debug if debug else LOG["debug"]

    def logf(self, filename=None):
        if filename is None:
            filename = LOG["file"]
        return "%s%s" % (LOG["dir"], filename)

    def get_logger(self, rotation=False):
        logger = logging.getLogger(self.logger_name)
        logf = self.logf(self.logfile)
        if rotation:
            from logging.handlers import TimedRotatingFileHandler
            self.handler = TimedRotatingFileHandler(logf, when='midnight', interval=1, backupCount=10)
        else:
            self.handler = logging.FileHandler(logf)
        formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s')
        #formatter = "%(asctime)s - %(levelname)s - %(filename)s:%(funcName)s - %(message)s",
        self.handler.setFormatter(formatter)
        logger.addHandler(self.handler)
        logger.setLevel(logging.DEBUG)
        return logger