Output colored logs from django through supervisor and docker-compose

Question

My objective is to get colorized logs of my Django webservice in docker-compose logs.

1. I use docker-compose to manage a list of web services based on Django framework.

2. Each container, run a my_init bash script, which in turn run a runit (this is historic in my case) script which runs a supervisord process:

   my_init---runsvdir-+-runsv---run---supervisord-+-gunicorn---gunicorn
                      |                           |-nginx---8*[nginx]
                      |                           |-python---9*[python]
                      |                           |-python---python  (Django)
                      |                           `-redis-server---2*[{redis-server}]
                      `-runsv
   

3. The Django server is interfaced in WSGI with a Gunicorn, and served through a Nginx

4. The supervisord conf is the following:

   [supervisord]
   http_port=/var/tmp/supervisor.sock ; (default is to run a UNIX domain socket server)
   stdout_logfile=/dev/stdout
   stdout_logfile_maxbytes=0
   stderr_logfile=/dev/stderr
   stderr_logfile_maxbytes=0
   
   [program:gunicorn_core_service]
   #environment=myapp_VENV=/opt/myapp/venv/
   environment=PYTHONPATH=/opt/myapp/myappServer/myappServer
   command = /opt/myapp/venv/bin/gunicorn wsgi -b 0.0.0.0:8000 --timeout 90 --access-logfile /dev/stdout --error-logfile /dev/stderr
   directory = /opt/myapp/myappServer
   user = root
   autostart=true
   autorestart=true
   redirect_stderr=true
   stdout_logfile=/dev/stdout
   stdout_logfile_maxbytes=0
   stderr_logfile=/dev/stderr
   stderr_logfile_maxbytes=0
   
   [program:django-celery]
   command=/opt/myapp/venv/bin/python ./manage.py celery --app=myappServer.celeryapp:app worker -B --loglevel=INFO
   directory=/opt/myapp/myappServer
   numprocs=1
   stdout_logfile=/dev/stdout
   stdout_logfile_maxbytes=0
   stderr_logfile=/dev/stderr
   stderr_logfile_maxbytes=0
   redirect_stderr=true
   autostart=true
   autorestart=true
   startsecs=10
   
   [program:nginx]
   command=nginx -g "daemon off;"
   #user = root
   autostart=true
   autorestart=true
   redirect_stderr=true
   stdout_logfile=/dev/stdout
   stdout_logfile_maxbytes=0
   stderr_logfile=/dev/stderr
   stderr_logfile_maxbytes=0
   

5. Since docker can only log once process, the logs of all the process of my container are forwarded to /dev/stdout and /dev/stderr

6. And I use colorlog as color formatter to colorize Django logs:

   'formatters': {
       'color': {
           '()': 'colorlog.ColoredFormatter',
           'format': '%(log_color)s%(levelname)-8s %(message)s',
           'log_colors': {
               'DEBUG':    'bold_black',
               'INFO':     'white',
               'WARNING':  'yellow',
               'ERROR':    'red',
               'CRITICAL': 'bold_red',
           },
       }
   },
   

   • When I run the service from inside the container

     ./manage.py runserver

     the logs are colorized.
   • However from outside, the docker-compose logs are not colorized.

I tried to add

tty: true

in my services as described here, but it seems it doesn\'t work anymore.

Any idea?

Answer 1

Here is a minimal working example for Django application served with Gunicorn. The trick is to make sure Gunicorn loggers are configured to use a handler that utilizes a colored formatter.

settings.py

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'colored_verbose': {
            '()': 'colorlog.ColoredFormatter',
            'format': "%(log_color)s%(levelname)-8s%(red)s%(module)-30s%(reset)s %(blue)s%(message)s"
        },
    },
    'handlers': {
        'colored_console': {
            'level': 'INFO',
            'class': 'logging.StreamHandler',
            'formatter': 'colored_verbose'
        }
    },
    'loggers': {
        '': {
            'level': 'INFO',
            'handlers': ['colored_console'],
        },
        'gunicorn.access': {
            'handlers': ['colored_console']
        },
        'gunicorn.error': {
            'handlers': ['colored_console']
        }
    }
}

Terminal output

Bonus

The best way (from my experience) to check your logging configuration is to use logging_tree.

Simply execute this in your application context (in Django shell for example)

import logging_tree
logging_tree.printout()

and it should print a nice representation of the existing loggers.

I went a step further and added this to my urls.py so I could see the logging configuration after all the loggers are installed.

from django.conf.urls import url
from django.http import HttpResponse
import logging_tree

urlpatterns = [
    url(r'^loggers', loggers),
]

def loggers(request):
    """
    Returns a representation of the existing loggers
    """
    return HttpResponse(logging_tree.format.build_description()[:-1])

It should return something similar to this

<--""
   Level INFO
   Handler Stream <_io.TextIOWrapper name='<stderr>' mode='w' encoding='UTF-8'>
     Level INFO
     Formatter <colorlog.colorlog.ColoredFormatter object at 0x1103a1668>
   |
   o<--"django"
   |   Level INFO
   |   Handler Stream <_io.TextIOWrapper name='<stderr>' mode='w' encoding='UTF-8'>
   |     Level INFO
   |     Filter <django.utils.log.RequireDebugTrue object at 0x11047d518>
   |   Handler <AdminEmailHandler (ERROR)>
   |     Level ERROR
   |     Filter <django.utils.log.RequireDebugFalse object at 0x11053a7f0>
   |   |
   |   o<--"django.db"
   |   |   Level NOTSET so inherits level INFO
   |   |   |
   |   |   o<--"django.db.backends"
   |   |       Level NOTSET so inherits level INFO
   |   |       |
   |   |       o<--"django.db.backends.schema"
   |   |           Level NOTSET so inherits level INFO
   |   |
   |   o<--"django.request"
   |   |   Level NOTSET so inherits level INFO
   |   |
   |   o   "django.server"
   |   |   Level INFO
   |   |   Propagate OFF
   |   |   Handler Stream <_io.TextIOWrapper name='<stderr>' mode='w' encoding='UTF-8'>
   |   |     Level INFO
   |   |     Formatter <django.utils.log.ServerFormatter object at 0x11053a630>
   |   |
   |   o<--"django.template"
   |       Level NOTSET so inherits level INFO
   |
   o<--"gunicorn"
       Level NOTSET so inherits level INFO
       |
       o   "gunicorn.access"
       |   Level INFO
       |   Propagate OFF
       |   Handler Stream <_io.TextIOWrapper name='<stderr>' mode='w' encoding='UTF-8'>
       |     Level INFO
       |     Formatter <colorlog.colorlog.ColoredFormatter object at 0x1103a1668>
       |
       o   "gunicorn.error"
       |   Level INFO
       |   Propagate OFF
       |   Handler Stream <_io.TextIOWrapper name='<stderr>' mode='w' encoding='UTF-8'>
       |     Level INFO
       |     Formatter <colorlog.colorlog.ColoredFormatter object at 0x1103a1668>
       |
       o<--"gunicorn.http"
          Level NOTSET so inherits level INFO
           |
           o<--"gunicorn.http.wsgi"
               Level NOTSET so inherits level INFO