Sunspot-Solr slowing down to a beast once my Application climbed to > 1000 objects [ Solr Logs Included ]

Question

I'm curious if anyone noticed any scaling issues with Sunspot-Solr . Even if I remove all the searchable params, and its just counting against the raw class by itself; it still takes 5 to 8 seconds to load on my local, 4 to 5 seconds on production.

Has anyone else been able to scale Sunspot-Solr ? What are some common problems?

How can one look into this deeper?

Here's the Solr Logs for a single request :

Solr Select (208.1ms)   {:rows=>20, :start=>0, :q=>"*:*", :sort=>"score desc", :fq=>["type:Organization", "published_b:true", "updated_at_d:[2009\\-02\\-03T16\\:11\\:55Z TO *]"]}

Solr Select (5.5ms)   {:rows=>20, :start=>0, :q=>"*:*", :sort=>"score desc", :fq=>["type:Organization", "published_b:true", "updated_at_d:[2009\\-02\\-03T16\\:11\\:55Z TO *]"]}

Solr Update (12.6ms)   <?xml version="1.0" encoding="UTF-8"?><add><doc><field name="type">User</field><field name="type">ActiveRecord::Base</field><field name="id">User 2</field><field name="class_name">User</field><field name="first_name_s">Bob</field><field name="created_at_d">2009-09-28T21:00:27Z</field><field name="last_name_s">Marley</field><field name="email_s">[email protected]</field><field name="name_s">Bob Marley</field><field name="last_name_text">Marley</field><field name="first_name_text">Bob</field><field name="email_text">[email protected]</field><field name="name_text">Bob Marley</field></doc></add>


Solr Update (487.7ms)   <?xml version="1.0" encoding="UTF-8"?><commit/>
Completed in 12632ms (View: 11633, DB: 228) | 200 OK [http://localhost/organizations/search]

Answer 1

1000 objects is child's play for Solr, so there's something fishy going on here with the ~200ms Solr read. However, your most immediate problem is that you're writing to Solr during what appears to be a GET request -- what's up with that? Are you saving a searchable object, which is triggering Sunspot's auto-index? If you have the need to update models during the course of a GET request (which should probably be done in a background job if possible), you'll want to disable auto-indexing in Sunspot:

searchable :auto_index => false
  # sunspot setup
end

Then you'd need to explicitly call my_model.index in your controllers when you actually do want to update them in Solr.

Finally, that big update at the end is a Solr commit, which tells Solr to write unstaged changes to disk and load up a new searcher that reflects those changes. Commits are expensive; Sunspot::Rails by default performs a commit at the end of any request that writes to Solr, but this behavior is targeted at the principle of least surprise for new users of Sunspot rather than a live app in production. You'll want to disable it in your config/sunspot.yml:

auto_commit_after_request: false

You'll then probably want to configure autoCommit in your solr/conf/solrconfig.xml -- it's commented out in the default Sunspot Solr distribution, and there is an explanation in there too. I've found that once a minute is a good place to start.

After making those changes, I'd see if your reads are still slow -- I think it's quite possible that the cause of that is that every time you search, your write/commit to Solr is causing it to have to load up a fresh searcher from disk. So, it can't allow any of its internal caches to warm, etc., and is generally under a tremendous amount of strain.

Hope that helps!

Answer 2

When I experienced long requests time during update commits I stumbled upon this blog

mytechmembank.blogspot.de

and it turned out that I had to change the following:

 Performance killer:
<str name="buildOnCommit">true</str>

Way to go:
<str name="buildOnCommit">false</str>

in solrconfig.xml