How to send celery all logs to a custom handler . in my case python-logstash handler

Question

In my Celery application I am getting 2 types of logs on the console i.e celery application logs and task level logs (inside task I am using logger.INFO(str) syntax for logging)

I wanted to send both of them to a custom handler (in my case python-logstash handler )

For django logs I was successfull, by setting handler and logger in settings.py but I am helpless with celery

Answer 1

def initialize_logstash(logger=None,loglevel=logging.DEBUG, **kwargs):
    # logger = logging.getLogger('celery')
    handler = logstash.TCPLogstashHandler('localhost', 5959,tags=['worker'])
    handler.setLevel(loglevel)
    logger.addHandler(handler)
    # logger.setLevel(logging.DEBUG)
    return logger

from celery.signals import after_setup_task_logger
after_setup_task_logger.connect(initialize_logstash)
from celery.signals import after_setup_logger
after_setup_logger.connect(initialize_logstash)

using both after_setup_task_logger and after_setup_logger signals solved the problem

Answer 2

Celery provides a after_setup_logger signal that is triggered after Celery has set up the logger. Among other few arguments, the signal passes the logger object which you can add your custom logging handlers to.

from celery import signals

import logstash
import logging


@signals.after_setup_logger.connect
def setup_logstash_logger(logger, *args, **kwargs):
    handler = logstash.TCPLogstashHandler('localhost', 5959)

    # More logger/handler configuration
    # handler.setLevel(logging.ERROR)
    # ...

    logger.addHandler(handler)

After fine-tuning Celery's logger you can simply rely on it to send the app messages to Logstash, even if you need to send your own messages:

logger = logging.getLogger(__name__)
logger.info('My message')  # This message will also be sent to Logstash