Solr suggester throws stackoverflow error

Question

Below error is being shown when using solr suggester. Has anyone faced similar error. Is the error due to the lookup factories limitation.

Error

{"error":{"msg":"java.lang.StackOverflowError",
 "trace":"java.lang.RuntimeException: java.lang.StackOverflowError
 org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:618)
 org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:477)
 org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:214)
 org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
 org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
 org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
 org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
 org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
 org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
 org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
 org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
 org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
 org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
 org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
 org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
 org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
 org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
 org.eclipse.jetty.server.Server.handle(Server.java:499)
 org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
 org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
 org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
 org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
 org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
 java.lang.Thread.run(Thread.java:745)\nCaused by: java.lang.StackOverflowError
 org.apache.lucene.util.automaton.Operations.topoSortStatesRecurse(Operations.java:1311)

Config Schema.xml

<field name="description" type="text_ja" 
       stored="true" indexed="true" multiValued="true"/>
<field name="textSuggest" type="textSuggest"
       indexed="true" stored="true" multiValued="true"/>

<copyField source="description" dest="text"/>

Solrconfig.xml

<searchComponent name="suggest" class="solr.SuggestComponent">
    <lst name="suggester">
      <str name="name">Suggester</str>
      <str name="lookupImpl">FuzzyLookupFactory</str>      
      <str name="dictionaryImpl">DocumentDictionaryFactory</str>
      <str name="field">textSuggest</str>
     <!-- <str name="weightField">price</str> -->
      <str name="suggestAnalyzerFieldType">string</str>
      <str name="buildOnStartup">false</str>
      <str name="buildOnCommit">false</str>
    </lst>
  </searchComponent>

  <requestHandler name="/suggest" class="solr.SearchHandler" 
                  startup="lazy" >
    <lst name="defaults">
      <str name="suggest">true</str>
      <str name="suggest.count">10</str>
    </lst>
    <arr name="components">
      <str>suggest</str>
    </arr>
  </requestHandler>

Answer 1

I had the same problem too: a customer asked me to activate the suggester module on a Win2008 machine with Solr 5.5.0, 1 core/node, 500k+ documents.

I suppose that the StackOverflowError is due to some FuzzyLookupFactory implementation that doesn't properly work when it has to build large Lookup Data Structure from scratch.

I was not able to make the suggester work with the FuzzyLookupFactory, in this situation the only solution for me was activating it using the FreeTextLookupFactory.

I post an excerpt of my configuration files, hope this helps:

solrconfig.xml

<searchComponent name="suggest" class="solr.SuggestComponent">
    <lst name="suggester">
        <str name="name">mySuggester</str>
        <str name="lookupImpl">FreeTextLookupFactory</str>
        <str name="dictionaryImpl">DocumentDictionaryFactory</str>
        <str name="field">content</str>
        <str name="suggestFreeTextAnalyzerFieldType">suggestTypeLc</str>
        <str name="buildOnStartup">true</str>
        <str name="buildOnCommit">false</str>
    </lst>
</searchComponent>

<requestHandler name="/suggest" class="solr.SearchHandler" startup="lazy">
    <lst name="defaults">
        <str name="suggest">true</str>
        <str name="suggest.count">10</str>
        <str name="suggest.dictionary">mySuggester</str>
    </lst>
    <arr name="components">
        <str>suggest</str>
    </arr>
</requestHandler>

schema.xml

<fieldType name="suggestTypeLc" class="solr.TextField" positionIncrementGap="100">
    <analyzer>
        <charFilter class="solr.PatternReplaceCharFilterFactory" pattern="[^a-zA-Z0-9]" replacement=" " />
        <tokenizer class="solr.WhitespaceTokenizerFactory"/>
        <filter class="solr.LowerCaseFilterFactory"/>
    </analyzer>
</fieldType>

More reference here and in the documentation here.

Please note:

1. buildOnCommit parameter was set to false for performances improvement
2. suggestTypeLc implements an analyzer that consider only alphanumeric/numeric character lowercase, although results are in original case

3. Even if the buildOnStartup parameter was set to true, I noticed that it doesn't seems to work. As first operation after reboot I have to manually perform a query with "suggest.build=true" parameter to make it really build. e.g.

   http://localhost:82/solr/mycore/suggest?suggest=true&suggest.build=true&suggest.dictionary=mySuggester&wt=json&suggest.q=docum
   
   In my configuration, this usually takes several minutes to complete. Following queries without force building are matter of milliseconds.