How to analyze memcache key hit rate with graphite?

Question

I have a Rails app which uses caching extensively and I want to know cache hit rate for different places in the app. Low hit rate places obviously need attention. But measure first!

To obtain real data I use graphite + statsd combo and a custom Dalli instrumentation employing statsd-instrument gem. All keys in the app are in form ['place', details...], so I get following metrics in graphite:

• stats.cache.place1.hits
• stats.cache.place1.misses
• stats.cache.place2.hits
• stats.cache.place2.misses
• etc.

Now I want to show all hit rates. I was able to come up with following formula for one place:

divideSeries(stats.cache.place1.hits, sumSeries(stats.cache.place1.*))

It works quite well, but there are several dozen places and I'd hate to duplicate it, not to mention that new places can appear.

This is a question for you, Graphite experts: is there a way to show hit rates for all places? I've seen group* functions in the docs, but they confuse me.

Ideally I want to segment my places into 4 categories:

• High hit rate, many requests. Caching is doing good job.
• Low hit rate, many requests. Needs attention.
• High hit rate, few requests. Is caching needed at all?
• Low hit rate, few requests. Definitely remove caching.

I'd be really grateful for any ideas how to use graphite for such analysis (I can request data in JSON format and do my own math, but I suspect there must be a simpler way).

Answer 1

You can use globs at multiple levels, so for the global view of how all caching is performing:

divideSeries(stats.cache.*.hits, sumSeries(stats.cache.*.*))

For the 4 categories you mention, the mostDeviant function might be good, that would help find the highest/lowest cache rates.

mostDeviant(5, divideSeries(stats.cache.*.hits, sumSeries(stats.cache.*.*)))

To group them into buckets based on requests and then showing a separate derived ratio is harder. Callback functions using repeated groupByNode and highestAverage might work

highestAverage(groupByNode(groupByNode(stats.cache.*.*, 3, "sumSeries"), 2, "divideSeries"), 10)

As a side note, with most LRU (least-recently used) cache eviction schemes, there isn't much point in removing caching, as it won't compete for cache space.