Grafana Prometheus Counter

Question

I have trying to get exact count for an event in a Grafana visualization using Prometheus as timeseries DB. But the counter is showing incorrect records. I am getting a higher count for 2 days that I am getting 7 days, which definately points to something being wrong.

First I have used a single stats visualization with this promql query:

sum(increase(http_server_requests_seconds_count[$__range])).

P.S.

I have also tried the following : sum(increase(http_server_requests_seconds_count[1m])) . This also gives incorrect counts.

I have tried the same with graph and using the legend to show totals in table. This is also gives incorrect counts.

Please let me know what is the best way of showing counts which can be reliable over time range changes.

My json :

{
    "colorMode": "value",
    "fieldOptions": {
        "calcs": [
            "lastNotNull"
        ],
        "defaults": {
            "mappings": [],
            "thresholds": {
                "mode": "absolute",
                "steps": [{
                    "color": "green",
                    "value": null
                }]
            }
        },
        "overrides": [],
        "values": false
    },
    "graphMode": "area",
    "justifyMode": "auto",
    "orientation": "auto"
},
"pluginVersion": "6.6.1",
"targets": [{
    "expr": " sum(increase(http_server_requests_seconds_count[$__range]))",
    "hide": false,
    "instant": true,
    "refId": "A"
}],
"timeFrom": null,
"timeShift": null,
"title": "Total Number of Requests",
"type": "stat"
}

Answer 1

This works for me:

sum(increase(http_request_duration_seconds_count{ecs_cluster=~"$ecs_cluster", instance_id=~"$instance_id"}[$__range]))

Activated instant query and set calculation to last not null

Here is the pane JSON:

{
  "cacheTimeout": null,
  "datasource": "Prometheus",
  "description": "",
  "fieldConfig": {
    "defaults": {
      "custom": {},
      "unit": " requests",
      "decimals": 0,
      "thresholds": {
        "mode": "absolute",
        "steps": [
          {
            "color": "blue",
            "value": null
          }
        ]
      },
      "mappings": [],
      "nullValueMode": "connected"
    },
    "overrides": []
  },
  "gridPos": {
    "h": 2,
    "w": 5,
    "x": 0,
    "y": 4
  },
  "id": 4,
  "interval": null,
  "links": [],
  "maxDataPoints": 100,
  "options": {
    "reduceOptions": {
      "values": false,
      "calcs": [
        "lastNotNull"
      ],
      "fields": ""
    },
    "orientation": "horizontal",
    "textMode": "auto",
    "colorMode": "value",
    "graphMode": "none",
    "justifyMode": "auto",
    "fieldOptions": {
      "calcs": [
        "lastNotNull"
      ]
    }
  },
  "pluginVersion": "7.1.0",
  "targets": [
    {
      "expr": "sum(increase(http_request_duration_seconds_count{ecs_cluster=~\"$ecs_cluster\", instance_id=~\"$instance_id\"}[$__range]))",
      "hide": false,
      "instant": true,
      "interval": "",
      "intervalFactor": 1,
      "legendFormat": "",
      "refId": "A"
    }
  ],
  "timeFrom": null,
  "timeShift": null,
  "title": "",
  "type": "stat"
}