rsyslog conditional RepeatedMsgReduction

Question

I need to deactivate RepeatedMsgReduction on my Linux server to allow fail2ban to evaluate every failed login attempt. (See this bug report: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=440037)

As I don't want to flood my log files, I would love to have that feature only on those logfiles that fail2ban scans, especially mail.warn.

Is there a way to set a conditional rule/filter in rsyslog.conf that sets

$RepeatedMsgReduction = off

iff the message is bound to mail.warn?

Answer 1

Ok, reading the manual and docs again and again now brought me to the solution:

Reading http://www.rsyslog.com/doc/rsconf1_repeatedmsgreduction.html tells you, that the directive

$RepeatedMsgReduction off

remains valid until the next directive is specified.

This means, if you want all messages of mail.warn, but want reduced messages in all others (mail.info and mail.err), change the configuration to look like this (assuming, that message reduction is globally turned on):

mail.info                       -/var/log/mail.info
$RepeatedMsgReduction off
mail.warn                       -/var/log/mail.warn
$RepeatedMsgReduction on
mail.err                        /var/log/mail.err

With this, mail.warn will contain all messages logged, while the other log levels (and files) will contain the famous "last message repeated n times" lines. This allows e.g. fail2ban to examine mail.warn for evil activities, while the other log files stay "clean".

Answer 2

I appreciate the research provided here showing how to selectively disable rsyslog message reduction. However, I think it important to recognize that there are arguments in favor of disabling reduction entirely.

Background:

Syslog's MARK facility is intended to provide evidence that syslog (and by inference the server itself) is running by periodically logging an entry as:

Sep 5 11:41:28 servername rsyslogd: -- MARK --

I came to this SO question because message reduction was defeating the purpose of MARK by aggregating my mark messages:

Sep 5 12:33:46 servername rsyslogd: message repeated 48 times: [-- MARK --]

I.e., the above message is logged in lieu of 48 periodic messages, nullifying their utility as tracking information.

Analysis:

Initially I therefore planned to disable reduction only for MARK messages. However, having read this article I am inclined to turn off RepeatedMsgReduction entirely:

• While turning this feature on can save some space in logs, most log analysis tools need to see the repeated messages, they can’t handle the “last message repeated” format.

• This is a feature that worked decades ago when logs were small and reviewed by a human, it fails badly on high volume logs processed by tools.

Conclusion:

I recommend simply disabling reduction. Why? Now that disk space is inexpensive and we rely increasingly on automated tools like fail2ban, the benefit of reduction is outweighed by the costs of configuration complexity and information loss.