Getting a true 95th percentile in DataDog

Question

I have an application that publishes a metric to DataDog with multiple tags, and my DataDog agent has a line that looks like

histogram_percentiles: 90, 95, 99

So my metric (lets call it ResponseTime) has a metric in the DataDog viewer for each of those (i.e. ResponseTime.90perentile).

However if you look at this metric carefully it appears to be calculating these percentiles on a short range (not sure what) and for each tuple of the tags that exist.

Ideally what I'd like to get is a 95th percentile of the ResponseTime metric over all the tags (maybe I filter it down by 1 or 2 and have a couple of different graphs) but over the last week or so. Is there an easy way to do this?

Answer 1

However if you look at this metric carefully it appears to be calculating these percentiles on a short range (not sure what) and for each tuple of the tags that exist.

The short range that you have noticed is actually the flush interval which defaults to 10 seconds. As per this article on histogram metric by datadog,

It aggregates the values that are sent during the flush interval (usually defaults to 10 seconds). So if you send 20 values for a metric during the flush interval, it'll give you the aggregation of those values for the flush interval

For your query -

Ideally what I'd like to get is a 95th percentile of the ResponseTime metric over all the tags (maybe I filter it down by 1 or 2 and have a couple of different graphs) but over the last week or so. Is there an easy way to do this?

as per my reading of the datadog docs, there isn't a way to get this done at the moment. It might be a good idea to check with datadog support regarding this.

More details here.