Speeding up Solr Indexing

Question

I am kind of working on speeding up my Solr Indexing speed. I just want to know by default how many threads(if any) does Solr use for indexing. Is there a way to increase/decrease that number.

Answer 1

When you index a document, several steps are performed :

• the document is analyzed,
• data is put in the RAM buffer,
• when the RAM buffer is full, data is flushed to a new segment on disk,
• if there are more than ${mergeFactor} segments, segments are merged.

The first two steps will be run in as many threads as you have clients sending data to Solr, so if you want Solr to run three threads for these steps, all you need is to send data to Solr from three threads.

You can configure the number of threads to use for the fourth step if you use a ConcurrentMergeScheduler (http://lucene.apache.org/java/3_0_1/api/core/org/apache/lucene/index/ConcurrentMergeScheduler.html). However, there is no mean to configure the maximum number of threads to use from Solr configuration files, so what you need is to write a custom class which call setMaxThreadCount in the constructor.

My experience is that the main ways to improve indexing speed with Solr are :

• buying faster hardware (especially I/O),
• sending data to Solr from several threads (as many threads as cores is a good start),
• using the Javabin format,
• using faster analyzers.

Although StreamingUpdateSolrServer looks interesting for improving indexing performance, it doesn't support the Javabin format. Since Javabin parsing is much faster than XML parsing, I got better performance by sending bulk updates (800 in my case, but with rather small documents) using CommonsHttpSolrServer and the Javabin format.

You can read http://wiki.apache.org/lucene-java/ImproveIndexingSpeed for further information.

Answer 2

This article describes an approach to scaling indexing with SolrCloud, Hadoop and Behemoth. This is for Solr 4.0 which hadn't been released at the time this question was originally posted.