Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to display values of metrics in alert rules prometheus

I create a rule in prometheus alertmanager, which tell about least space on mountpoints in percents - and additional to this i want to show how much least space in gigabytes , but i do not want to hardcoded a mountpoint to show gigabytes, i want use $labels.mountpoint which come from a "expr" , not hardcode.

i find a similar problem on this link https://github.com/prometheus/alertmanager/issues/549 but in this case using hardcoded mountpoint

this is my rule

- alert: OutOfDiskSpace
    expr: node_filesystem_free_bytes / node_filesystem_size_bytes * 100 < 10
    for: 1m
    labels:
      severity: Critical
    annotations:
      description: "Disk is almost full (< 10% left)\n {{ $labels.instance_short }}\n {{ $labels.mountpoint }}\n VALUE = {{ printf `node_filesystem_avail_bytes / 1024 / 1024 / 1024` | query | first | value | humanize }}"

when i used node_filesystem_avail_bytes / 1024 / 1024 / 1024 in VALUE, i do not take the mountpoint from expression, but i know where actual value, - it's in $labels.mountpoint which i can't use in template or don't know how to do this

like image 724
Sergei Cherevko Avatar asked Oct 15 '22 12:10

Sergei Cherevko


1 Answers

  - alert: OutOfDiskSpace
      expr: node_filesystem_free_bytes / node_filesystem_size_bytes * 100 < 10
      for: 5s
      labels:
        severity: Critical
      annotations:
        description: "Disk is almost full (< 10% left)\n {{ $labels.instance_short }}\n {{ $labels.mountpoint }}\n VALUE = {{ printf \"node_filesystem_avail_bytes{mountpoint='%s'}\" .Labels.mountpoint | query | first | value | humanize1024 }}"
like image 174
Sergei Cherevko Avatar answered Oct 21 '22 07:10

Sergei Cherevko