Trying to debug issue with logback syslog appender not updating syslog

Question

I am using logback to update syslog, this is how i configured my appender:

<appender name="SYSLOG" class="ch.qos.logback.classic.net.SyslogAppender">
        <syslogHost>localhost</syslogHost>
        <facility>LOCAL0</facility>
        <suffixPattern>[%thread] %logger %msg</suffixPattern>
    </appender>

I updated rsyslog.conf to listen for UDP events, uncommented the below lines:

# Provides UDP syslog reception
$ModLoad imudp.so
$UDPServerRun 514

Restarted syslog daemon after conf changes.

On all my test boxes, it seems to be working just fine! However, one on system syslog is not being updated by my process (Other stuff is updating it just fine), i'm wondering how I could go about debugging this problem? Anything I should look into that comes to mind?

Thanks for any ideas

Answer 1

Sure. Loosely in sequence, try these 4 tests:

1. Test logback: The obvious one: add a FileAppender as a second appender and make sure events appear there. Your post implies that "it" works in dev but I wasn't sure whether it is logback or the appender, and the config snippet doesn't have an appender-ref section to send events to SYSLOG.

If your FileAppender receives nothing, it's an app/environment problem or this server is not generating events that feed to the appender.

2. Confirm messages are being generated: Assuming the FileAppender receives messages but syslog doesn't, run:

tcpdump -n -i lo -X -s 1500

.. to output the full payload of UDP packets on lo. Make your app generate a log message. You should see at least 1 packet destined to 127.0.0.1:514. If you don't, it's the sender. If you do, it's the rsyslog config.

3. Confirm that rsyslog is bound to port 514:

lsof -i :514

or if you don't have lsof and are sure that another process isn't bound to 514:

netstat -ln | grep 514

4. See what rsyslog receives: If events are being sent to port 514, after stopping the live rsyslogd, restart it in debug mode and attached to the terminal:

/etc/init.d/rsyslog stop
rsyslogd -d

You should see events arriving. If none of those identify the problem, it's something way off the beaten path. I've got a working logback config on known-good J2EE and syslog environments. Hopefully one of the things above will do it, though.