Sunspot-solr is eating up too much memory? What's a convenient way to reduce memory usage?

Question

So it seems Sunspot-Solr is eating a lot of memory up. And its probably due to my malpractice as a programmer allocating proper usage of it.

First of all, I do not allow sunspot to reindex by itself.

searchable :auto_index => false do

This alone I believe is preventing it from reindexing. Instead I run the reindex process on a cron-tab that runs once a day in the early morning.

The reason I started doing this, was because as the application scaled, the data seemed to sitting so heavily on the machine that it would take quite awhile just to simply load the homepage.

What am I doing wrong to cause the error below? And what can I do better?

Error in question:

ActionView::TemplateError (Solr Response: Lock_obtain_timed_out_NativeFSLocktmpindexlucenede61b2c77401967646cf8916982a09a0writelock__orgapachelucenestoreLockObtainFailedException_Lock_obtain_timed_out_NativeFSLocktmpindexlucenede61b2c77401967646cf8916982a09a0writelock__at_orgapachelucenestoreLockobtainLockjava85__at_orgapacheluceneindexIndexWriterinitIndexWriterjava1545__at_orgapacheluceneindexIndexWriterinitIndexWriterjava1402__at_orgapachesolrupdateSolrIndexWriterinitSolrIndexWriterjava190__at_orgapachesolrupdateUpdateHandlercreateMainIndexWriterUpdateHandlerjava98__at_orgapachesolrupdateDirectUpdateHandler2openWriterDirectUpdateHandler2java173__at_orgapachesolrupdateDirectUpdateHandler2addDocDirectUpdateHandler2java220__at_orgapachesolrupdateprocessorRunUpdateProcessorprocessAddRunUpdateProcessorFactoryjava61__at_orgapachesolrhandlerXMLLoaderprocessUpdateXMLLoaderjava139__at_orgapachesolrhandlerXMLLoaderloadXMLLoaderjava69__at_orgapachesolrhandlerContentStreamHandlerBasehandleRequestBodyContentStreamHandlerBasejava54__at_orgapachesolrhandlerRequestHandlerBasehandleRequestRequestHandlerBasejava131__at_orgapachesolrcoreSolrCoreexecuteSolrCorejava1316__at_orgapachesolrservletSolrDispatchFilterexecuteSolrDispatchFilterjava338__at_orgapachesolrservletSolrDispatchFilterdoFilterSolrDispatchFilterjava241__at_orgmortbayjettyservletServletHandler$CachedChaindoFilterServletHandlerjava1089__at_orgmortbayjettyservletServletHandlerhandleServletHandlerjava365__at_orgmortbayjettysecuritySecurityHandlerhandleSecurityHandlerjava216__at_orgmortbayjettyservletSessionHandlerhandleSessionHandlerjava181__at_orgmortbayjettyhandlerContextHandlerhandleContextHandlerjava712__at_orgmortbayjettywebappWebAppContexthandleWebAppContextjava405__at_orgmortbayjettyhandlerContextHandlerCollectionhandleContextHandlerCollectionjava211__at_orgmortbayjettyhandlerHand) on line #24 of app/views/main/_main_nav.html.haml:
21:         %br
22:         community calendars
23: 
24:   - if (current_user.blank? || current_user.card_signup.blank?)
25:     %li
26:       - if current_user.blank?
27:         = link_to 'Get Your HQcard', signup_path, :title => "Signup for your free HQcard and redeem local deals and promotions."

    rsolr (0.12.1) [v] lib/rsolr/connection/requestable.rb:39:in `request'
    rsolr (0.12.1) [v] lib/rsolr/client.rb:34:in `request_without_rails_logging'
    /usr/lib/ruby/gems/1.8/gems/sunspot_rails-1.2.1/lib/sunspot/rails/solr_logging.rb:25:in `request'
    /usr/lib/ruby/gems/1.8/gems/sunspot_rails-1.2.1/lib/sunspot/rails/solr_logging.rb:24:in `request'
    rsolr (0.12.1) [v] lib/rsolr/client.rb:22:in `update'
    rsolr (0.12.1) [v] lib/rsolr/client.rb:46:in `add'
    sunspot (1.2.1) lib/sunspot/indexer.rb:101:in `add_documents'
    sunspot (1.2.1) lib/sunspot/indexer.rb:26:in `add'
    sunspot (1.2.1) lib/sunspot/session.rb:91:in `index'
    sunspot (1.2.1) lib/sunspot/session_proxy/abstract_session_proxy.rb:11:in `index'
    sunspot (1.2.1) lib/sunspot.rb:175:in `index'
    /usr/lib/ruby/gems/1.8/gems/sunspot_rails-1.2.1/lib/sunspot/rails/searchable.rb:349:in `solr_index'
    /usr/lib/ruby/gems/1.8/gems/sunspot_rails-1.2.1/lib/sunspot/rails/searchable.rb:405:in `maybe_auto_index'
    vendor/gems/binarylogic-authlogic-2.1.1/lib/authlogic/acts_as_authentic/session_maintenance.rb:73:in `save_without_session_maintenance'
    vendor/gems/binarylogic-authlogic-2.1.1/lib/authlogic/session/callbacks.rb:83:in `save_record'
    vendor/gems/binarylogic-authlogic-2.1.1/lib/authlogic/session/priority_record.rb:30:in `save_record'
    vendor/gems/binarylogic-authlogic-2.1.1/lib/authlogic/session/persistence.rb:60:in `persisting?'
    vendor/gems/binarylogic-authlogic-2.1.1/lib/authlogic/session/persistence.rb:39:in `find'
    app/controllers/application_controller.rb:23:in `current_user_session'
    app/controllers/application_controller.rb:28:in `current_user'
    (eval):2:in `send'
    (eval):2:in `current_user'
    app/views/main/_main_nav.html.haml:24:in `_run_haml_app47views47main47_main_nav46html46haml_locals_main_nav_object'
    haml (2.2.2) [v] lib/haml/helpers/action_view_mods.rb:11:in `render'
    haml (2.2.2) [v] lib/haml/helpers.rb:96:in `non_haml'
    haml (2.2.2) [v] lib/haml/helpers/action_view_mods.rb:11:in `render'
    app/views/main/index.html.haml:2:in `_run_haml_app47views47main47index46html46haml'
    haml (2.2.2) [v] lib/haml/helpers/action_view_mods.rb:13:in `render'
    haml (2.2.2) [v] lib/haml/helpers/action_view_mods.rb:13:in `render'
    haml (2.2.2) [v] rails/./lib/sass/plugin/rails.rb:19:in `process'
    lib/flash_session_cookie_middleware.rb:14:in `call'
    vendor/gems/hoptoad_notifier-2.2.2/lib/hoptoad_notifier/rack.rb:27:in `call'

Answer 1

You can specify minimum and maximum memory allocation as per the current environment in your config/sunspot.yml file, e.g.

development:
  solr:
    min_memory: 512M
    max_memory: 1G
    solr_jar: /path/to/start.jar
    log_level: DEBUG

Documentation: http://outoftime.github.com/sunspot/rails/docs/classes/Sunspot/Rails/Configuration.html