Django Celery Task Logging

Question

I have setup Celery in a Django project on which I am working. I would like to separate the logging for celery tasks vs the remainder of celery logs (celerycam, celerybeat, etc).

Based on the Celery documentation (http://docs.celeryproject.org/en/latest/userguide/tasks.html#logging) it seems like I should just be able to define a Django logger for 'celery.task' which should do this. However, when I do this, nothing shows up in the logs. Everything does show up if I create a generic 'celery' logger, implying that this may be something to do with the name of the logger.

What am I missing here? Is there a way to make this work, or must all celery logs go together?

For what it's worth, I have set CELERYD_HIJACK_ROOT_LOGGER = False.

My logging setup in settings.py:

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'filters': {
        'require_debug_false': {
            '()': 'django.utils.log.RequireDebugFalse'
        }
    },
    'formatters': {
        'standard': {
            'format': '%(asctime)s %(levelname)s [%(name)s: %(lineno)s] -- %(message)s',
            'datefmt': '%m-%d-%Y %H:%M:%S'
        },
    },
    'handlers': {
        'logfile': {
            'level': 'INFO',
            'filters': None,
            'class': 'logging.handlers.RotatingFileHandler',
            'filename': '/vagrant/logs/logfile.log',
            'maxBytes': 1024*1024*5,
            'backupCount': 3,
            'formatter': 'standard'
        },
        'debug_logfile': {
            'level': 'DEBUG',
            'filters': None,
            'class': 'logging.handlers.RotatingFileHandler',
            'filename': '/vagrant/logs/debug_logfile.log',
            'maxBytes': 1024*1024*5,
            'backupCount': 5,
            'formatter': 'standard'
        },
        'default_logger': {
            'level': 'WARNING',
            'filters': None,
            'class': 'logging.handlers.RotatingFileHandler',
            'filename': '/vagrant/logs/default.log',
            'maxBytes': 1024*1024*5,
            'backupCount': 2,
            'formatter': 'standard'
        },
        'celery_logger': {
            'level': 'DEBUG',
            'filters': None,
            'class': 'logging.handlers.RotatingFileHandler',
            'filename': '/vagrant/logs/celery.log',
            'maxBytes': 1024*1024*5,
            'backupCount': 2,
            'formatter': 'standard'
        },
        'celery_task_logger': {
            'level': 'DEBUG',
            'filters': None,
            'class': 'logging.handlers.RotatingFileHandler',
            'filename': '/vagrant/logs/celery_tasks.log',
            'maxBytes': 1024*1024*5,
            'backupCount': 2,
            'formatter': 'standard'
        },
    },
    'loggers': {
        '': {
            'handlers': ['default_logger'],
            'level': 'WARNING',
            'propagate': True,
        },
        'django': {
            'handlers': ['logfile'],
            'level': 'INFO',
            'propagate': True,
        },
        'feedmanager': {
            'handlers': ['logfile', 'debug_logfile'],
            'level': 'DEBUG',
            'propagate': True,
        },
        'recipemanager': {
            'handlers': ['logfile', 'debug_logfile'],
            'level': 'DEBUG',
            'propagate': True,
        },
        'menumanager': {
            'handlers': ['logfile', 'debug_logfile'],
            'level': 'DEBUG',
            'propagate': True,
        },
        'celery.task': {
            'handlers': ['celery_task_logger'],
            'level': 'DEBUG',
            'propagate': True,
        },
        'celery': {
            'handlers': ['celery_logger'],
            'level': 'DEBUG',
            'propagate': True,
        },
    }
}

Answer 1

i'm guessing in your tasks your doing this

from celery.utils.log import get_task_logger
logger = get_task_logger(__name__)

if your doing that then your logger name will be the module name, so if your task is in a tasks.py file in an app called MyApp your logger will be named 'MyApp.tasks' and you'd have to create the logger 'MyApp.tasks' in your settings.

You can just put a different string in place of __name__ for all of the tasks to log to same logger if you have them all over place. ie: 'celery.task'

oh and make sure your worker loglevel is to what you want it to be