Prometheus Collector fails with "collected metric was collected before with the same name and label values"

Question

I have a device that is exposing temperature measurements as a JSON in the following format:

[
  {
    "dataPointId": 123456,
    "values": [
      {
        "t": 1589236277000,
        "v": 14.999993896484398
      },
      {
        "t": 1589236877000,
        "v": 14.700006103515648
      },
      {
        "t": 1589237477000,
        "v": 14.999993896484398
      },
[..]

As you can see, the values contain both a timestamp and the temperature measurement. I would like to expose these measurements via Prometheus metrics, so I am using prometheus/client_golang to build an exporter.

My expectation would be that the /metrics endpoint then exposes something like this from the data above:

# HELP my_temperature_celsius Temperature
# TYPE my_temperature_celsius gauge
my_temperature_celsius{id="123456"} 14.999993896484398 1589236277000
my_temperature_celsius{id="123456"} 14.700006103515648 1589236877000
my_temperature_celsius{id="123456"} 14.999993896484398 1589237477000

I implemented a simple prometheus.Collector and I am adding my static metrics without any issues. For the measurements above, NewMetricWithTimestamp seems to be the only way to add metrics with a timestamp, so I am iterating over these values using something like this:

for _, measurements := range dp.Values {
  ch <- prometheus.NewMetricWithTimestamp(
    time.Unix(measurements.T, 0),
    prometheus.MustNewConstMetric(
      collector.temperature,
      prometheus.GaugeValue,
      float64(measurements.V),
      device.DatapointID))
}

However, this leads to the following error that I do not fully understand:

An error has occurred while serving metrics:

1135 error(s) occurred:
* collected metric "my_temperature_celsius" { label:<name:"id" value:"123456" > gauge:<value:14.999993896484398 > timestamp_ms:1589236877000000 } was collected before with the same name and label values
* collected metric "my_temperature_celsius" { label:<name:"id" value:"123456" > gauge:<value:14.700006103515648 > timestamp_ms:1589237477000000 } was collected before with the same name and label values
[..]

• I understand that the metric and label combination must be unique, but as I am also adding a timestamp, does that not count as a unique metric? Is my expectation above even possible?

• How can I represent these measurements in a Prometheus exporter?

Answer 1

Ref from Prometheus

A gauge is a metric that represents a single numerical value that can arbitrarily go up and down.

A histogram samples observations (usually things like request durations or response sizes) and counts them in configurable buckets. 

Gauge used for one value that we care about, don't care about timestamp. Like Current temperature, not tempratures of last day.

Gauge is not the metric type you are looking for. Or, prometheus may not what you are looking for.

When we want to monitor temprature, we use histogram. which you can calc average temp, min temp or max in a short time. BUT, when you want to use your own timestamp, you need to implement A histogram collector yourself. You can check the file from prometheus/client_golang/histogram.go. Not simple AT ALL.

What you really need is A time series database, like influxdb. You can push you data into influxdb which accept custom timestamp, as simple as post json to http, and then monitor data with grafana.

Wish that would help you.

Answer 2

If you look closely, then you'll see the JSON data format is slightly redundant in context of metric collection, because the timestamps are inside each device rather than being a parent key and having values as an array of device IDs and values. Only then would you be looping over real time series data, and then your labels won't be static over a loop, like they are now. Label uniqueness is label name + label value hashed together.

I think then the preferred approach would be to make a Gauge Vector. Use WithLabelValues to get a Gauge object and call Set on it to set the value

deviceTempGaugeVector := prometheus.NewGaugeVec(
    prometheus.GaugeOpts{
        Name: "my_temperature_celsius",
    },
    []string{
        "device_id" // Using single label instead of 2 labels "id" and "value"
    },
)

prometheus.MustRegister(deviceTempGaugeVector)

for _, point := range dp.TimeStamps {
  for _, measurements := range point {
    deviceId := measurements.DatapointID
    value := measurements.V
    metric := deviceTempGaugeVector.WithLabelValues(deviceId).Set(value)
    ch <- prometheus.NewMetricWithTimestamp(time.Unix(measurements.T, 0),metric)
  }
}

Ref : https://godoc.org/github.com/prometheus/client_golang/prometheus#NewGaugeVec