grafana: Alerting when sum of values in a day is less than half the sum from the day one week ago

Question

I have a metric foo.bar that is incremented (+1) many times a day. The number of times the metric is incremented across a day is x. I want to detect whether there is something terribly wrong by alerting when x on the most recent full 24 hour period is less than half of x* from the same weekday 7 days prior.

What alert can I use for this?

Answer 1

You could try alerting on something like:

divideSeries(hitcount(foo.bar,"1day"),hitcount(timeShift(foo.bar, "7d"), "1day"))

And set an alert to fire if that value drops below 0.5. That will work best, I think, if you run the alert in a Grafana view with a time window some amount shorter than 1 day (this is a dim hunch, so take it with a grain).

If hitcount doesn't process the data correctly, you could use integral (or some foo.bar.total value if you have aggregation set up in Graphite itself). If you use integral, though, beware the accuracy-related pitfalls discussed in this article. That writeup also discusses using integral(hitcount(...)), but since hitcount already aggregates, I don't think that will meet your use case. I may be wrong though.