OutOfMemory with Solr4

Question

I'm indexing the content I have and after upgrade my Solr instance to solr 4 I'm facing some OutOfMemories. The exception thrown is:

INFO  org.apache.solr.update.UpdateHandler - start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
ERROR o.a.solr.servlet.SolrDispatchFilter - null:java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space
        at org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:469)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:297)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:240)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:164)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:164)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:562)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:395)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:250)
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:188)
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:166)
        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:302)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.OutOfMemoryError: Java heap space

Is there some known bug or something I could test to get rid of it?

Within this upgrade two things changed:

• solr version (from 3.4 to 4.0);
• lucene match version (from LUCENE_34 to LUCENE_40).

Answer 1

Seems to be running out of memory when accessing logs, at a glance. That may not be particularly meaningful, with an 'Out of Memory' error, of course, but worth a shot, particularly after seeing this complaint regarding SOLR 4.0 logging. Particularly so if this is occuring during an index rebuild of some form, or heavy load of updates.

So try disabling the update log, which I believe can be done by commenting out:

<updateLog> 
  <str name="dir">${solr.data.dir:}</str> 
</updateLog> 

in solrconfig.xml.

EDIT:

Another (possibly better) approach to this, taking another glance at it, might be to commit more often. The growth of the update log seems to be directly related to having a lot of queued updates waiting for commit.

If you do not have autocommit enabled, you might want to try adding it in your config, something like:

 <autoCommit> 
   <maxTime>15000</maxTime> 
   <openSearcher>false</openSearcher> 
 </autoCommit> 

There's a good bit of related discussion and recommendation to be found on this thread.