We have prometheus running on Win Server box, and WMI exporter on a separate box(client). Able to read client metrics in Prometheus. Now the requirement is the moment Diskspace =>90 % , send an email alert, so that we can run a job to clean up space using an automated job / manual job.
Could you please help on how to configure alert for diskspace >90
Setting up alerts with Prometheus is a two-step process: To start, you need to create your alerting rules in Prometheus, and specify under what conditions you want to be alerted (such as when an instance is down). Second, you need to set up Alertmanager, which receives the alerts specified in Prometheus.
The Alertmanager handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integration such as email, PagerDuty, or OpsGenie. It also takes care of silencing and inhibition of alerts.
assuming you are using https://github.com/martinlindhe/wmi_exporter/blob/master/docs/collector.logical_disk.md you could use something along these lines for > 90 % use
- alert: DiskSpaceUsage
expr: 100.0 - 100 * (wmi_logical_disk_free_bytes / wmi_logical_disk_size_bytes) > 90
for: 10m
labels:
severity: high
annotations:
summary: "Disk Space Usage (instance {{ $labels.instance }})"
description: "Disk Space on Drive is used more than 90%\n VALUE = {{ $value }}\n LABELS: {{ $labels }}"
there are other examples on wmi_exporter repo for default node_exporter metrics ( not sure if available with windows ) it should be
- alert: DiskSpace10%Free
expr: 100 - (100 * node_filesystem_avail_bytes / node_filesystem_size_bytes) > 90
labels:
severity: moderate
annotations:
summary: "Instance {{ $labels.instance }} is low on disk space"
description: "diskspace on {{ $labels.instance }} is used over {{ $value }}% ."
´´´
You might want to alert based on if it's going to fill up, not based on how full it is:
- name: node.rules
rules:
- alert: DiskWillFillIn4Hours
expr: predict_linear(node_filesystem_free{job="node"}[1h], 4 * 3600) < 0
for: 5m
labels:
severity: page
https://www.robustperception.io/reduce-noise-from-disk-space-alerts
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With