Django, Haystack, Solr, MongoDB architecture decision

Question

I'm building a log viewing utility which will handle real-time search in TBs of logs. I have decided to store logs in Solr and use it as search engine.I will use Django as framework in my project. In order to use Solr with Django I saw there is haystack. Now my architecture will be like this.

             Store  Index         Search             Show
Log Stream ----------------> Solr --------> Haystack ------> Django

My logs are ordinary linux server logs, like network, operational, error, etc. Syslog is sending logs. I will allow filtering based on all log line. I will allow sorting by columns, for ex: ip column, date column etc.

Example log:

Dec 11 13:24:03 2012 [firewall] R0 SRC=192.168.9.11 DST=192.168.11.29 LEN=83 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=36904 DPT=161 LEN=63 

I want to ask is it better to use MongoDB as storage, filtering and search in logs or Solr will do it better. Elasticsearch is coming in my mind, too. What will be your choice in such case.

Thanks in advance.

Answer 1

Why reinventing the wheel? There's Logstash, with an amazing interface: Kibana. You can feed it using rsyslog. However, if you really want/need to reimplement a log server, Logstash uses ElasticSearch. I would go with it.