Python SysLogHandler over TCP: handling connection loss

Question

I have a process sending logs to a syslog server over TCP using logging.SyslogHandler. Unfortunately, if the syslog server is restarted for some reason, the process stops sending logs and is unable to re-establish the connection.

I was wondering if anyone knows of a way to overcome this behaviour and force logging.SyslogHandler to re-establish the connection.

Code to use the handler would be something like:

import logging
import logging.handlers
import logging.config

logging.config.fileConfig('logging.cfg')
logging.debug("debug log message")

logging.cfg:

[loggers]
keys=root,remote

[handlers]
keys=local,remote

[logger_remote]
qualname=remote
level=INFO
handlers=remote

[logger_root]
qualname=root
level=DEBUG
handlers=local

[handler_local]
class=handlers.StreamHandler
level=DEBUG
formatter=local
args=(sys.stdout,)

[handler_remote]
class=handlers.SysLogHandler
level=DEBUG
formatter=remote
args=(('localhost', 514), handlers.SysLogHandler.LOG_USER, 1)

[formatters]
keys=local,remote

[formatter_local]
format=%(module)s %(levelname)s %(message)s

[formatter_remote]
format=%(asctime)s %(message)s

The error I keep getting after the syslog server restarts is:

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/logging/handlers.py", line 866, in emit
    self.socket.sendall(msg)
  File "/usr/local/lib/python2.7/socket.py", line 228, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 32] Broken pipe

I would appreciate any insights. Thanks!

Answer 1

I ran into this same issue. I had to write a custom handler that handles broken pipe exceptions and re-creates the socket.

class ReconnectingSysLogHandler(logging.handlers.SysLogHandler):                
    """Syslog handler that reconnects if the socket closes                      

    If we're writing to syslog with TCP and syslog restarts, the old TCP socket 
    will no longer be writeable and we'll get a socket.error of type 32.  When  
    than happens, use the default error handling, but also try to reconnect to  
    the same host/port used before.  Also make 1 attempt to re-send the         
    message.                                                                    
    """                                                                         
    def __init__(self, *args, **kwargs):                                        
        super(ReconnectingSysLogHandler, self).__init__(*args, **kwargs)        
        self._is_retry = False                                                  

    def _reconnect(self):                                                       
        """Make a new socket that is the same as the old one"""                 
        # close the existing socket before getting a new one to the same host/port
        if self.socket:                                                         
            self.socket.close()                                                 

        # cut/pasted from logging.handlers.SysLogHandler                        
        if self.unixsocket:                                                     
            self._connect_unixsocket(self.address)                              
        else:                                                                   
            self.socket = socket.socket(socket.AF_INET, self.socktype)          
            if self.socktype == socket.SOCK_STREAM:                             
                self.socket.connect(self.address)                               

    def handleError(self, record):                                              
        # use the default error handling (writes an error message to stderr)    
        super(ReconnectingSysLogHandler, self).handleError(record)              

        # If we get an error within a retry, just return.  We don't want an     
        # infinite, recursive loop telling us something is broken.              
        # This leaves the socket broken.                                        
        if self._is_retry:                                                      
            return                                                              

        # Set the retry flag and begin deciding if this is a closed socket, and  
        # trying to reconnect.                                                  
        self._is_retry = True                                                   
        try:                                                                    
            __, exception, __ = sys.exc_info()                                  
            # If the error is a broken pipe exception (32), get a new socket.   
            if isinstance(exception, socket.error) and exception.errno == 32:   
                try:                                                            
                    self._reconnect()                                           
                except:                                                         
                    # If reconnecting fails, give up.                           
                    pass                                                        
                else:                                                           
                    # Make an effort to rescue the recod.                       
                    self.emit(record)                                           
        finally:                                                                
            self._is_retry = False