simple Solr deployment with two servers for redundancy

Question

I'm deploying the Apache Solr web app in two redundant Tomcat 6 servers, to provide redundancy and improved availability. At this point, scalability is not a issue.

I have a load balancer that can dynamically route traffic to one server or the other or both.

I know that Solr supports master/slave configuration, but that requires manual recovery if the slave receives updates during the master outage (which it will in my use case).

I'm considering a simpler approach using the ability to reload a core: - only one of the two servers is receiving traffic at any time (the "active" instance), but both are running, - both instances share the same index data and - before re-routing traffic due to an outage, the now active instance is told to reload the index core(s)

Limited testing of failovers with both index reads and writes has been successful. What implications/issues am I missing?

Your thoughts and opinions welcomed.

Answer 1

The simple approach to redundancy your considering seems reasonable but you will not be able to use it for disaster recovery unless you can share the data/index to/from a different physical location using your NAS/SAN.

Here are some suggestions:-

1. Make backups for disaster recovery and test those backups work as an index could conceivably have been corrupted as there are no checksums happening internally in SOLR/Lucene. An index could get wiped or some records could get deleted and merged away without you knowing it and backups can be useful for recovering those records/docs at a later time if you need to perform an investigation.

2. Before you re-route traffic to the second instance I would run some queries to load caches and also to test and confirm the current index works before it goes online.

3. Isolate the updates to one location and process and thread to ensure transactional integrity in the event of a cutover as it could be difficult to manage consistency as SOLR does not use a vector clock to synchronize updates like some databases. I personally would keep a copy of all updates in order separately from SOLR in some other store just in case a small time window needs to be repeated.

In general, my experience with SOLR has been excellent as long as you are not using cutting edge features and plugins. I have one instance that currently has 40 million docs and an uptime of well over a year with no issues. That doesn't mean you wont have issues but gives you an idea of how stable it could be.