log4j -adaptive log level- feature

Question

Don't want to reinvent the wheel so I am wondering if any logging system already supports something like what I am proposing to do.

Background: I am working on an extremely large system where tens of thousands of users access the servers at any given time. There are a lot of surrounding infrastructures involved so you can imagine what it looks like to investigate rare occurring bugs just by reading the logs in such an ecosystem.

Our system uses log4j.

The main root of the problem was that the original implementors were rather 'economical', to put it mildly, when encountering an unknown error. Sure, the DEBUG level would be helpful in most cases, but of course, production logs are set on ERROR level only :(

Now I am trying to make a better world for our children and want to expand the logging system to be more 'forgiving' about the log level.

What I have in mind goes like this:
Since the surrounding DEBUG level logs (N before and M after the ERROR log) might contain important data, why not lower the log level of the system around the ERROR.

My idea:

1. let's assume ERROR log level set for the system
2. use some kind of a rollover logging buffer of size X. Every log goes in there and is forwarded to log4j.
3. When buffer gets an ERROR log it can also output the context of N previous and M next log messages by either
   1. artificially raising the log level to ERROR state (and noting that, of course) or
   2. just log that context via an injected ERROR log

There are some issues to work out but in general it could work.

Any ideas?

Cheers

Answer 1

BufferingForwardingAppender might be a solution. The documentation associated with the example states "This means that the events will only be delivered when a message with level of WARN or higher level is logged. Up to 512 (BufferSize) previous messages of any level will also be delivered to provide context information. Messages not sent will be discarded."

See: http://logging.apache.org/log4net/release/config-examples.html#buffering

For busy web applications this is probably not the best approach. In most cases you are interested in the log messages associated with the specific session or request that caused the ERROR. Since one request is usually handled by a single thread, this could be achieved by keeping a "per thread" buffer for the log messages. By adding a servlet filter one could then clear the buffer everytime the server starts processing new request from client.

Or even better would be to keep (if memory usage is not an issue) the buffers per HTTPSession. This way you could actually get the debugging info not just for the request that caused issues, but also for previous request (to help figuring out what the user was doing)