Dynamically remove duplicate log messages

Question

Recently we had a message fill up /var/log/libvirt/qemu/.log in a matter of minutes with a line that repeated that crashed our system due to the root partition being filled (20+ Gigs in minutes).

"block I/O error in device 'drive-virtio-disk0': Operation not permitted (1)"

Is there a way to ensure that duplicate lines are not pushed into logs, or a way to limit that directory from filling up? Logstash maxsize will not work for us since we run it on a daily cronjob.

Answer 1

It depends on which logging utility you are using (rsyslog or syslog-ng)

Rsyslog can remove repeated messages by adding lines like: "last message repeated 3044 times". To enable this option you should add:

$RepeatedMsgReduction on

to /etc/rsyslog.conf

I don't know if such reduction is possible with syslog-ng.

Both syslog-ng and rsyslog can completely remove lines matching some pattern:

• rsyslog - take a look into this manual: http://www.rsyslog.com/discarding-unwanted-messages/

• syslog-ng - take a look in filters. there is some example how to do it: https://serverfault.com/questions/540038/excluding-some-messages-from-syslog-ng