Performance comparison for logging to MSMQ / text file / database

Question

We have several choices for logging in our .NET (C#) server application. We are going to use Enterprise Library. So here are the ways to go:

1) Writing log to MSMQ synchronously, then reading MSMQ with Win Service. Queue is on the local machine for server application.
2) Writing log to disk (i.e. rolling text files) synchronously.
3) Writing log to database (Oracle, in our application) synchronously.

Log amount might be fairly high. So which one is the most performant? I guess ordering is 1, 2, 3. Am I right? Is there any other performance factor, besides write speed, in this specific scenario? Are there any other choices, that I have not pointed out here, that might be more better way?

Answer 1

Without hearing the application's needs in terms of logging:

I get the feeling that the MSMQ logging scenario is a bit of a large hammer to the problem.

MSMQ definitely will be faster, as your application will be using a network communication protocol instead of dealing with disk latency and DB connection creation & tear down. Given though, that MSMQ uses the disk, the gain is likely negligible. Then the question is whether that MSMQ target/destination queue is on the local machine.

Stick with the Disk Logging

Consider sticking with logging to a file on disk with appropriate rollovers (day/hour as required). There are some workarounds if you're SUPER concerned with spindle speeds - consider writing those log files to a RAM disk. Have then a process to copy those files out of RAM and onto disk periodically as you see fit.

Measure Your Logging Performance Needs

Cue the jokes about premature optimization. Measure, and measure again. You just won't know what you need until you gauge how well your solution is performing. Consider that you likely have 10,000rpm spindles, and likely a disk logging solution will perform adequately. A bit of YAGNI might creep in if the decision is to jump to the complex solution of an 'outsourced' component like MSMQ or a database write. I say this with your stated needs around speed.

Answer 2

I think you should consider "Write log to database asynchronously". i.e. for each item you want to log, queue it up in memory and have a separate thread writing to the database. No sense keeping the main application waiting for logging to complete provided the order of the logging messages is maintained by the queuing mechanism.

Also, did you look at log4net? It includes numerous logging options including a powerful rotating log to disk, UDP logging, ....

You might also want to look at Splunk as a way to search your log files after making them.