I have a Solr setup. One master and two slaves for replication. We have about 70 Millions documents in index. The slaves have 16 GBs of RAM. 10GBs for the OS and HD, 6GBs for Solr.
But from time to time, the slaves are out of memory. When we downloaded the dump file just before it was out of memory, we could see that the class :
org.apache.solr.util.ConcurrentLRUCache$Stats @ 0x6eac8fb88
is using up to 5Gb of memory. We are using filter caches extensively, it has a 93% hit ratio. And here's the xml for the filter cache in solrconfig.xml
<property name="filterCache.size" value="2000" />
<property name="filterCache.initialSize" value="1000" />
<property name="filterCache.autowarmCount" value="20" />
<filterCache class="solr.FastLRUCache"
size="${filterCache.size}"
initialSize="${filterCache.initialSize}"
autowarmCount="${filterCache.autowarmCount}"/>
The query results have the same settings, but is using the LRUCache and it only uses about 35mb of the memory. Is there something wrong with the configuration which needs to be fixed, or do I just need more memory for the filter cache?
The most typical way Solr uses the filterCache is to cache results of each fq search parameter, though there are some other cases as well. Subsequent queries using the same parameter filter query result in cache hits and rapid returns of results.
The query cache — Stores sets of document IDs returned by queries. If your velvet pants query returns 1,000 results, a set of 1,000 document IDs (integers) will be stored in the query cache for that query string. The filter cache — Stores the filters built by Solr in response to filters added to queries.
If you want to disable a cache (or all), comment out those sections in solrconfig. xml and restart solr.
In solrconfig. xml, the 'autowarmCount' of filterCache(or queryResultCache and etc) indicates how many cache entity will be copied when a new searcher come. However, if I add or delete doc in solr, there may be invalid cache entity exists in old searcher.
After a friend told me how roughly the filter caches works, it become clear why we get out of memory errors from time to time.
So what does the filter cache do? Basically it creates something like a bit array which tell which documents matched the filter. Some something like:
cache = [1, 0, 0, 1, .. 0]
1 means it hits, and 0 means no hit. So for the example, it means the filter cache matches the 0th and 3rd documents. So a cache is kind of like an array of bit, with the length of the total documents. So let's say I have 50 millions docs, so the array length will be 50 millions, which means one filter cache will take up 50.000.000 bit in the ram.
So we specified we want 2000 filter cache, it means the RAM it will take is roughly:
50.000.000 * 2000 = 100.000.000.000 bit
If you convert it to Gb. It will be:
100.000.000.000 bit / 8 (to byte) / 1000 (to kb) / 1000 (to mb) / 1000 (to gb) = 12,5 Gb
So the total RAM needed just by the filter cache is roughly 12Gb. And it means if the Solr only have 6Gb Heap Space, it will not be able to create 2000 filter caches.
Yes, I know Solr doesn't always create this array, and if the result of the filter query is low, it can just create something else which take up less memory. This calculation just says roughly how much the upper limit of the filter cache is, if it has 2000 caches in the ram. It can be lower in other better cases.
So one solution is to lower the number of maximum filter caches in solr config. We checked solr stats, most of the time we only have about 600 filter caches, so we can reduce the filter caches number to that as the maximum.
Another option is to of course add more RAM.
Some options:
If when querying, sometimes you know the fq will be very rare, dont cache it, by using
fq={!cache=false}inStock:true
of course, get more memory is another option
investigate if DocValues help here, they do help with memory in other scenarios (facetting, sorting...), but not sure if they do with fq
if you are not at latest release, upgrade.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With