Getting facet count 0 in solr

Question

I am using solr search with faceting in my application. My use case is in such a way that the index files in the datadir keeps on changing.

The problem is, when I facet based on a particular field. I get the value from the indices that where previously in the data dir (and are not present currently). However they are returned with a value of 0. I don't understand where the values from the previous indices are persisted and are returned during a totally newer search?

Though I can simply skip the facets with count 0, I understand that this can seriously eat over my scalability. Any pointers to not include the facets from previous searchers?

[Edit 1] : The current workaround I am using is add a facet.mincount=1 in my URL. But still, I guess this can eat over my performance.

Answer 1

I couldnt find a comment option & I dont have enough reputation to vote-up! I have the same exact problem. We are using atomic updates with solr 4.2.

I found some explanation here: http://collab.sakaiproject.org/pipermail/oae-dev/2011-November/000693.html

Excerpt:

To efficiently handle facets for multi-valued fields (like tags), Solr builds an "uninverted index" (which you think would just be called an "index", but I suppose that's even more confusing), which maps internal document IDs to the list of terms they contain. Calculating facets from this data structure just requires walking over every document in the result set, looking up the terms it contains in the uninverted index, and adding them to the tally for all documents.

However, there's a sneaky optimisation here that causes the zero counts we're seeing. For terms that appear in more than 5% of documents, Solr doesn't include them in the uninverted index (leaving them out helps to keep the size in memory down, I guess), and instead gets the count for these terms using a regular query against the Lucene index. Since the set of "common" terms isn't specific to your result set, and since any given result set won't necessarily contain all of these terms, you can get back counts of zero.

It may not be from old index values but just terms that exist in more than 5% of documents?

Answer 2

I think facet.mincount=n is not a workaround, you should use it to get only the non-negative facet count.

solrQuery.setQuery("*:*");
solrQuery.addFacetField("foobar");
solrQuery.setFacetMinCount(1);