How to monitor the size of a directory via Telegraf

Question

We need to monitor the size of a directory (for example the data directory of InfluxDB) to set up alerts in Grafana. As mentioned here: How to configure telegraf to send a folder-size to influxDB , there is no built-in plugin for this.

We don't mind using the inputs.exec section of Telegraf. The directories are not huge (low filecount + dircount), so deep scanning (like the use of du) is fine by us.

One of the directories we need to monitor is /var/lib/influxdb/data.

What would be a simple script to execute, and what are the caveats?

Answer 1

You could create a simple bash script metrics-exec_du.sh with the following content (chmod 755):

#!/usr/bin/env bash
du -bs "${1}" | awk '{print "[ { \"bytes\": "$1", \"dudir\": \""$2"\" } ]";}'

And activate it by putting the following in the Telegraf config file:

[[inputs.exec]] commands = [ "YOUR_PATH/metrics-exec_du.sh /var/lib/influxdb/data" ] timeout = "5s" name_override = "du" name_suffix = "" data_format = "json" tag_keys = [ "dudir" ]

Caveats:

1. The du command can stress your server, so use with care
2. The user telegraf must be able to scan the dirs. There are several options, but since InfluxDB's directory mask is a bit unspecified (see: https://github.com/influxdata/influxdb/issues/5171#issuecomment-306419800), we applied a rather crude workaround (examples are for Ubuntu 16.04.2 LTS):
   • Add the influxdb group to the user telegraf : sudo usermod --groups influxdb --append telegraf
   • Put the following in the crontab, run for example each 10 minutes: 10 * * * * chmod -R g+rX /var/lib/influxdb/data > /var/log/influxdb/chmodfix.log 2>&1

Result, configured in Grafana (data source: InfluxDB):

Cheers, TW