How do I setup logging when using aiohttp and aiopg with Gunicorn?

Question

aiohttp is great, but setting up logging has been a nightmare, both locally and in production, when using Gunicorn.

Most of the examples and documentation I find for setting up logging are for running in native server mode, where you use make_handler()

As recommended in the documentation, I'm using Gunicorn as a Web Server to deploy, so I don't call make_handler explicitly.

I am not seeing aiohttp.access logs, nor the aiohttp.server logs, nor the aiopg logs, all of which should be set up by default

This is what I've got in a root level app.py:

import logging

import aiopg
from aiohttp import web

async def some_handler(request):
    id = request.match_info["id"]
    # perform some SA query
    return web.json_response({"foo": id})

async def close_postgres(app):
    app['postgres'].close()
    await app['postgres'].wait_closed

async def init(loop, logger, config):
    app = web.Application(
        loop=loop,
        logger=logger
    )

    app['postgres'] = await aiopg.sa.create_engine(loop=loop, echo=True) # other args ommitted
    app.on_cleanup.append(close_postgres)

    app.router.add_route('GET', '/', some_handler, 'name')

    return app

def run():
    config = parse_yaml('config.yml') # => turns config.yml to dict
    logging.config.dictConfig(config['logging'])
    logger = logging.getLogger("api")

    loop = asyncio.get_event_loop()
    app = run_until_complete(init(loop, logger, config))

    return app

My config.yml file

logging:
  version: 1
  formatters:
    simple:
      format: '[%(asctime)s] [%(process)d] [%(levelname)s] %(message)s'
      datefmt: '%Y-%m-%d %H:%M:%S %z'
  handlers:
    console:
      class: logging.StreamHandler
      formatter: simple
      level: DEBUG
      stream: ext://sys.stdout
  loggers:
    api:
      handlers:
        - console
      level: DEBUG

I launch gunicorn with the following:

gunicorn 'app:run()' --worker-class aiohttp.worker.GunicornWebWorker

I only see the following logs no matter what query I make:

[2016-08-22 11:26:46 -0400] [41993] [INFO] Starting gunicorn 19.6.0
[2016-08-22 11:26:46 -0400] [41993] [INFO] Listening at: http://127.0.0.1:8000 (41993)
[2016-08-22 11:26:46 -0400] [41993] [INFO] Using worker: aiohttp.worker.GunicornWebWorker
[2016-08-22 11:26:46 -0400] [41996] [INFO] Booting worker with pid: 41996

What I want:

• aiopg logs (which queries ran)
• access logs
• server logs

Thanks

Answer 1

Documentation don't ultimately recommend to use Gunicorn for deployment but have instructions for running under Gunicorn.

Perhaps it should be upgraded to passing correct format for access logger.

From my perspective the easiest way to run aiohttp server is just running it (by using web.run_app() handler or building own runner on top of it).

If you need several aiohttp instances -- use nginx in reverse proxy mode (most likely you already have it in your tool chain) and supervisord for controlling servers.

The combination just works without the need for intermediate layer. Just like people starts tornado or twisted.