How to enable rotation of log files for celeryd with djcelery?

Question

I have below settings in my celery configuration file:

1. CELERYD_LOG_DEBUG='FALSE'
2. CELERYD_LOG_FILE=r'/var/log/celery/celeryd.log'
3. CELERYD_LOG_LEVEL="ERROR"

I looked at the settings file of the celery and there is no option to specify the log file size limit in the configuration. Even the code looks like its using the StreamHandler instead of the RotationHandler class. Any clues/hints ?

Thanks.

Answer 1

For log rotation on Ubuntu, if you have your celery log in /var/log/celery/ celeryd.log you can use a configuration like the one above to do weekly log rotation with /etc/logrotate.d/.

/var/log/celery/*.log {
    weekly
    missingok
    rotate 52
    compress
    delaycompress
    notifempty
    copytruncate
}

Answer 2

Try using a [WatchedFileHandler][1]. It notices that a file has been truncated or changed otherwise and reopens the file. Note you must set CELERY_HIJACK_ROOT_LOGGER = False and then setup this logger yourself, and do logging.getLogger('custom_logger') inside the tasks yourself. See this post about how Celery's builtin logging configuration is not flexible and for fine-grained control.