How do you manage logging performance?

Question

We have a message processing system where low latency is critical. Recently, I found that while we keep a high rate through our system we are seeing some "outliers." (Messages that take much longer then they should) When we removed logging our systems show none of these outliers.

Right now our logging is basically just a wrapped ostream with some logging-level functionality similar to log4j (debug, fatal, debug, etc).

I was wondering, what do others do to manage logging performance, specifically in message processing activities? How do you manage these I/O bound activities? Do you stripe it out? Do you move to databases instead?

Any advice for optimizing logging is appreciated.

Note: I recognize that there might be other problems with our system that causes the outliers, but for the sake of this question I am only interested in logging optimizations, thanks.

Also: Logging is mandatory for our system.

Answer 1

I guess it's OS dependent to some extent.

On win32, our logging subsystem simply queues the messages up for a logging thread which handles the disk I/O.

This decouples disk I/O performance from time-critical threads, and gives us good control over exactly how and when the queue gets locked.