Twisted logging

Question

I have 3 processes running under my twisted reactor: Orbited, WSGI (running django), and Twisted itself.

I am currently using

log.startLogging(sys.stdout)

When all the log are directed to the same place, there is too much flooding.

One line of my log from WSGI is like this:

2010-08-16 02:21:12-0500 [-] 127.0.0.1 - - [16/Aug/2010:07:21:11 +0000] "GET /statics/js/monitor_rooms.js HTTP/1.1" 304 - "http://localhost:11111/chat/monitor_rooms" "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.8) Gecko/20100723 Ubuntu/10.04 (lucid) Firefox/3.6.8"

The time is repeated twice basically. I think I should use my own formatter but unfortunately I cannot find it in twisted's docs (there's nothing on logging there)

1. What's the best way to deal with logging from 3 sources?
2. What kwargs do I pass in to which function in twisted.log to set up my own formatter (startLogging doesn't contain the answer)
3. What is a better solution than what I have suggested? ( I am not really experienced in setting up loggers. )

Answer 1

You can use the system keyword argument to twisted.python.log.msg to customize the message.

Assuming you've got:

log.msg("Service ready for eBusiness!", system="enterprise")

You'll get logging output like this:

2010-08-16 02:21:12-0500 [enterprise] Service ready for eBusiness!

You could then have each of your services add system="wsgi/orbited/..." to their log.msg and log.err calls.

I found this digging through the source last time I was working with Twisted.

Answer 2

Heh. I am thinking about exactly this problem. What I've come up with is a separate Twisted app that logs messages it receives over a socket. You can configure Python logging to send to a socket and you can configure Twisted's logging to send to Python logging. So you can get everything to send logging messages to a single process (which then uses Python's logging to log them to disk).

I have some initial proof of concept code at http://www.acooke.org/cute/APythonLog0.html

The main thing missing is that it would be nice to indicate which message came from which source. Not sure how best to add that yet (one approach would be to run the service on three different ports and have a different prefix for each).

PS How's the Orbited working out? That's on my list next...