Adding Per request Context to Logging in Python

Question

Background

For a REST api service, I'd like to provide more logging context in a way that I don't have to rewrite the log statements for my entire application. I'm using the python logging lib within flask and the eventlet runner type with gunicorn.

Use Case

Imagine a future where all requests through this system have a unique(enough) transaction ID passed as a header from some upstream source (reverse proxy maybe). I'd like to log this transaction id with each log statement to make it easy to trace a given request through my system even during peak load.

Approach

Write a custom logging context filter class that pulls desired information from flask. It is my understanding that I should be able to pull this info (namely the request object) from thread local context variables. Upon initialization of the global root logger, then I simply set this custom context filter and all should be good in the land of debugging!

I discovered this approach from the following cookbook documentation... https://docs.python.org/2/howto/logging-cookbook.html#using-filters-to-impart-contextual-information

Questions

• Do you foresee any scaling issues with this approach?
• Thoughts on propagate this transaction id downstream to other requests across my network?
• Will the use of the eventlet worker type get in the way of this behaving as expected (i.e. mixed context as a result of concurrency issues)?
• Just because you can, doesn't mean you should. Any other reason why I shouldn't do it this way?

Answer 1

It sounds like you may be considering implementing an X-Trace like system. Disclaimer: I work with a commercial product which operates like this.

Regarding propagation it can become a concern as the complexity of the app grows. As you require more off the shelf solutions or polyglot components in your back end these will require support for context logging or be burdened by a lack of visibility. Same goes for alternate messaging mechanisms, if you need to pass context over something besides HTTP-RPC like JBoss or Thrift or a message queue of some kind implementation complexity goes up.

For async requests you will definitely need to ensure the unique id is passed correctly from the blocking code path into the evented code path or risk mixing / re-using id's. This can be the source of subtle red herrings when analyzing call propagation.