Solr "real time" indexing

Question

I know there are several questions similar to this but they don't provide a simple answer to the problem at hand. Sorry if you feel this is a duplicate but I think clear and understandable answer would benefit many. So, to the question.

Can Solr indexing updates be automated? And if they can, what would be the optimal way to do it?

Here is a simple use case to clarify the question: I have a database table with several columns of different kind of data. There is a web app which is used to manage the data. I've got separate Solr server to index specified columns in the above mentioned table. How could I achieve an outcome that when users adds, removes or modifies data in the said table, Solr would notice the changed and modify the index.

It would be necessary for it to be "real time". Meaning that after few seconds the changes would take place. Of course with large amount of data it can be more.

Thanks in advance

Answer 1

There are two questions here:

Can Solr indexing updates be automated?

Yes they can, and they should be always automated. You don't want to manually launch the indexing process for every change.

It would be necessary for it to be "real time".

I already mentioned some ways to reduce latency between changed data and updating the index in this answer. You could use autoCommit to make sure that your data is committed within x seconds of the update. Depending on the interval, you'd want to reduce autowarming and adjust other settings, see this for more details.

Also keep an eye on the NRT wiki page for related information and solutions about this.

Answer 2

You may want to take a look at Apache Solr 3.3 with RankingAlgorithm 1.2. It supports NRT (Near Real Time Indexing) and can update 10,000 docs / sec. You can concurrently search during the updates. You do not need to commit or close the searchers. You can get more information about NRT with Solr 3.3 with RankingAlgorithm from here:

http://solr-ra.tgels.org/wiki/en/Near_Real_Time_Search_ver_3.x