Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to configure alerts in Prometheus for diskspace

We have prometheus running on Win Server box, and WMI exporter on a separate box(client). Able to read client metrics in Prometheus. Now the requirement is the moment Diskspace =>90 % , send an email alert, so that we can run a job to clean up space using an automated job / manual job.

Could you please help on how to configure alert for diskspace >90

enter image description here

like image 213
Abraham Dhanyaraj Avatar asked Oct 16 '18 20:10

Abraham Dhanyaraj


People also ask

How does Prometheus define alerts?

Setting up alerts with Prometheus is a two-step process: To start, you need to create your alerting rules in Prometheus, and specify under what conditions you want to be alerted (such as when an instance is down). Second, you need to set up Alertmanager, which receives the alerts specified in Prometheus.

How does Prometheus Alert Manager work?

The Alertmanager handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integration such as email, PagerDuty, or OpsGenie. It also takes care of silencing and inhibition of alerts.


2 Answers

assuming you are using https://github.com/martinlindhe/wmi_exporter/blob/master/docs/collector.logical_disk.md you could use something along these lines for > 90 % use

  - alert: DiskSpaceUsage
expr: 100.0 - 100 * (wmi_logical_disk_free_bytes / wmi_logical_disk_size_bytes) > 90
for: 10m
labels:
  severity: high
annotations:
  summary: "Disk Space Usage (instance {{ $labels.instance }})"
  description: "Disk Space on Drive is used more than 90%\n  VALUE = {{ $value }}\n  LABELS: {{ $labels }}"

there are other examples on wmi_exporter repo for default node_exporter metrics ( not sure if available with windows ) it should be

- alert: DiskSpace10%Free
     expr: 100 - (100 * node_filesystem_avail_bytes / node_filesystem_size_bytes) > 90
     labels:
       severity: moderate
     annotations:
       summary: "Instance {{ $labels.instance }} is low on disk space"
       description: "diskspace on {{ $labels.instance }} is used over {{ $value }}% ."
´´´
like image 94
Karsten Avatar answered Sep 17 '22 19:09

Karsten


You might want to alert based on if it's going to fill up, not based on how full it is:

- name: node.rules
  rules:
  - alert: DiskWillFillIn4Hours
    expr: predict_linear(node_filesystem_free{job="node"}[1h], 4 * 3600) < 0
    for: 5m
    labels:
      severity: page

https://www.robustperception.io/reduce-noise-from-disk-space-alerts

like image 35
ptman Avatar answered Sep 17 '22 19:09

ptman