figuring out reason for maxClauseCount is set to 1024 error

Question

I've two sets of search indexes. TestIndex (used in our test environment) and ProdIndex(used in PRODUCTION environment). Lucene search query: +date:[20090410184806 TO 20091007184806] works fine for test index but gives this error message for Prod index.

"maxClauseCount is set to 1024"

If I execute following line just before executing search query, then I do not get this error. BooleanQuery.SetMaxClauseCount(Int16.MaxValue); searcher.Search(myQuery, collector);

Am I missing something here? Why am not getting this error in test index?The schema for two indexes are same.They only differ wrt to number of records/data.PROD index has got higher number of records(around 1300) than those in test one (around 950).

Answer 1

The range query essentially gets transformed to a boolean query with one clause for every possible value, ORed together.

For example, the query +price:[10 to 13] is tranformed to a boolean query

+(price:10 price:11 price:12 price:13)

assuming all the values 10-13 exist in the index.

I suppose, all of your 1300 values fall in the range you have given. So, boolean query has 1300 clauses, which is higher than the default value of 1024. In the test index, the limit of 1024 is not reached as there are only 950 values.

Answer 2

I had the same problem. My solution was to catch BooleanQuery.TooManyClauses and dynamically increase maxClauseCount.

Here is some code that is similar to what I have in production.

private static Hits searchIndex(Searcher searcher, Query query) throws IOException
{
    boolean retry = true;
    while (retry)
    {
        try
        {
            retry = false;
            Hits myHits = searcher.search(query);
            return myHits;
        }
        catch (BooleanQuery.TooManyClauses e)
        {
            // Double the number of boolean queries allowed.
            // The default is in org.apache.lucene.search.BooleanQuery and is 1024.
            String defaultQueries = Integer.toString(BooleanQuery.getMaxClauseCount());
            int oldQueries = Integer.parseInt(System.getProperty("org.apache.lucene.maxClauseCount", defaultQueries));
            int newQueries = oldQueries * 2;
            log.error("Too many hits for query: " + oldQueries + ".  Increasing to " + newQueries, e);
            System.setProperty("org.apache.lucene.maxClauseCount", Integer.toString(newQueries));
            BooleanQuery.setMaxClauseCount(newQueries);
            retry = true;
        }
    }
}

Answer 3

I had this same issue in C# code running with the Sitecore web content management system. I used Randy's answer above, but was not able to use the System get and set property functionality. Instead I retrieved the current count, incremented it, and set it back. Worked great!

catch (BooleanQuery.TooManyClauses e)
{
    // Increment the number of boolean queries allowed.
    // The default is 1024.
    var currMaxClause = BooleanQuery.GetMaxClauseCount();
    var newMaxClause = currMaxClause + 1024;
    BooleanQuery.SetMaxClauseCount(newMaxClause);
    retry = true;
}