Send log messages from all celery tasks to a single file

Question

Note: This answer is outdated as of Celery 3.0, where you now use get_task_logger() to get your per-task logger set up. Please see the Logging section of the What's new in Celery 3.0 document for details.

---

Celery has dedicated support for logging, per task. See the Task documentation on the subject:

You can use the workers logger to add diagnostic output to the worker log:

@celery.task()
def add(x, y):
    logger = add.get_logger()
    logger.info("Adding %s + %s" % (x, y))
    return x + y

There are several logging levels available, and the workers loglevel setting decides whether or not they will be written to the log file.

Of course, you can also simply use print as anything written to standard out/-err will be written to the log file as well.

Under the hood this is all still the standard python logging module. You can set the CELERYD_HIJACK_ROOT_LOGGER option to False to allow your own logging setup to work, otherwise Celery will configure the handling for you.

However, for tasks, the .get_logger() call does allow you to set up a separate log file per individual task. Simply pass in a logfile argument and it'll route log messages to that separate file:

@celery.task()
def add(x, y):
    logger = add.get_logger(logfile='tasks.log')
    logger.info("Adding %s + %s" % (x, y))
    return x + y 

Last but not least, you can just configure your top-level package in the python logging module and give it a file handler of it's own. I'd set this up using the celery.signals.after_setup_task_logger signal; here I assume all your modules live in a package called foo.tasks (as in foo.tasks.email and foo.tasks.scaling):

from celery.signals import after_setup_task_logger
import logging

def foo_tasks_setup_logging(**kw):
    logger = logging.getLogger('foo.tasks')
    if not logger.handlers:
        handler = logging.FileHandler('tasks.log')
        formatter = logging.Formatter(logging.BASIC_FORMAT) # you may want to customize this.
        handler.setFormatter(formatter)
        logger.addHandler(handler)
        logger.propagate = False

after_setup_task_logger.connect(foo_tasks_setup_logging)

Now any logger whose name starts with foo.tasks will have all it's messages sent to tasks.log instead of to the root logger (which doesn't see any of these messages because .propagate is False).

Just a hint: Celery has its own logging handler:

from celery.utils.log import get_task_logger
logger = get_task_logger(__name__)

Also, Celery logs all output from the task. More details at Celery docs for Task Logging