Storing and Analyzing Logs Database Selection

Question

I am building an internal tool, which will be open-sourced, to take logs and put them into a database - to put it simply. From there, the tool will also analyze the logs and help alert the sys-admins and developers of issues going on, all in real-time. This is a lot of CPU to process this, more than the scope of this question.

What I would like to know is what Database to choose that will allow and perform quickly a number of key tasks:

• Store a large number of events categorized by event types
• Perform a large number of reads to develop charts to analyze the events that are being logged
• Read in real-time to send and trigger automated alerts to the system.

And any other help would be greatly appreciated, too. Code On.

Answer 1

To my observation MongoDB performs in a magnitude better than RDBS for a task you describe - massive store of logs. Particularly good performers are capped collections. Major performance lag with RDBS I've seen was the insert times. Huge disadvantage of RDBS is the schema which is a major pain to upgrade if needed. Because of these reasons we have started to move towards MongoDB - check out logFaces. If you are building your own tool for the open source community - try to make sure it will work with ANY database, not just a particular brand. But then it becomes a not so trivial task :)

(for disclosure - I am the original author of logFaces, so the opinion could be biased)