Searching through elmah error log files(Perhaps in 1000's)

Question

We have a folder of elmah error logs in XML format. These files will be in millions and each file might be upto 50 kb in size. We need to be able to search on the files(eg: What errors occured, what system failed etc). Do we have a open source system that will index the files and perhaps help us search through the files using keywords? I have looked at Lucene.net but it seems that I will have the code the application. Please advise.

Answer 1

If you need to have the logs in a folder in XML, elmah-loganalyzer might be of use.

You can also use Microsoft's Log Parser to perform "sql like" queries over the xml files:

LogParser -i:XML "SELECT * FROM *.xml WHERE detail like '%something%'"

EDIT:

You could use a combination of nutch+SOLR or logstash+Elastic Search as an indexing solution.

• http://wiki.apache.org/nutch/NutchTutorial
• http://lucene.apache.org/solr/tutorial.html

• http://blog.building-blocks.com/building-a-search-engine-with-nutch-and-solr-in-10-minutes

• http://www.logstash.net/

• http://www.elasticsearch.org/tutorials/using-elasticsearch-for-logs/
• http://www.javacodegeeks.com/2013/02/your-logs-are-your-data-logstash-elasticsearch.html