How to log data from multiple threads?

Question

There are huge numbers of threads running in parallel continuously (let's assume this continuous part)). All the threads want to log some application data, basically a set of values.

1. What would be the best approach to log this data? single/multiple file?
2. What would be the best approach to make backup of this log?
3. What would be the approach to read data from backup file and convert it into something useful?

Several threads like this and this suggest log4net and log4j but I want to know the actual process? Also how multiple threads write to same log file? Is file level lock required for each thread? How does all this work?

Any pointer towards understanding all the details would be appreciated.

Thanks.

Answer 1

A library like log4j will be able to be configured for your needs.

1. Splitting into too many files will make it difficult to debug some issues, but having one monolithic file leaves a soup of mixed processes. I would have a file for each atomic process, that is, a mail manager might use its own log file. Extra debug information for jdbc might have its own log file, but errors and major events would still be reported in the main application log.

2. Major logging libraries support log splitting and rotation. For a well used web application, I prefer to have a log file made for each day, and also split over a certain size. You can build a cron to zip older logs and depending on the application, you may want to back them up for a few months or indefinitely.

3. As far as debugging usefulness, you can grep for certain strings such as "Exception" to report on. If you are looking for statistics, you should make a log for that specific purpose in addition to your process log.

Logs can be synchronous or asynchronous, and the latter is usually best for performance. In general, a queue of messages is built and then written by a separate thread. So multiple threads can write to that one queue or buffer in memory and one thread will lock and write the file. Its pretty much in the background and you don't have to think about it unless you are writing a huge amount of data.