Robust endless loop for server written in Python

Question

I write a server which handles events and uncaught exceptions during handling the event must not terminate the server.

The server is a single non-threaded python process.

I want to terminate on these errors types:

• KeyboardInterrupt
• MemoryError
• ...

The list of built in exceptions is long: https://docs.python.org/2/library/exceptions.html

I don't want to re-invent this exception handling, since I guess it was done several times before.

How to proceed?

1. Have a white-list: A list of exceptions which are ok and processing the next event is the right choice
2. Have a black-list: A list of exceptions which indicate that terminating the server is the right choice.

Hint: This question is not about running a unix daemon in background. It is not about double fork and not about redirecting stdin/stdout :-)

Answer 1

I would do this in a similar way you're thinking of, using the 'you shall not pass' Gandalf exception handler except Exception to catch all non-system-exiting exceptions while creating a black-listed set of exceptions that should pass and end be re-raised.

Using the Gandalf handler will make sure GeneratorExit, SystemExit and KeyboardInterrupt (all system-exiting exceptions) pass and terminate the program if no other handlers are present higher in the call stack. Here is where you can check with type(e) that a __class__ of a caught exception e actually belongs in the set of black-listed exceptions and re-raise it.

As a small demonstration:

import exceptions  # Py2.x only

# dictionary holding {exception_name: exception_class}
excptDict = vars(exceptions)

exceptionNames = ['MemoryError', 'OSError', 'SystemError'] # and others

# set containing black-listed exceptions
blackSet = {excptDict[exception] for exception in exceptionNames}

Now blackSet = {OSError, SystemError, MemoryError} holding the classes of the non-system-exiting exceptions we want to not handle.

A try-except block can now look like this:

try:
    # calls that raise exceptions:
except Exception as e:
    if type(e) in blackSet: raise e # re-raise
    # else just handle it

An example which catches all exceptions using BaseException can help illustrate what I mean. (this is done for demonstration purposes only, in order to see how this raising will eventually terminate your program). Do note: I'm not suggesting you use BaseException; I'm using it in order to demonstrate what exception will actually 'pass through' and cause termination (i.e everything that BaseException catches):

for i, j in excptDict.iteritems():
    if i.startswith('__'): continue  # __doc__ and other dunders
    try:
        try:
            raise j
        except Exception as ex:
            # print "Handler 'Exception' caught " + str(i)
            if type(ex) in blackSet:
                raise ex           
    except BaseException:
        print "Handler 'BaseException' caught " + str(i)

# prints exceptions that would cause the system to exit     
Handler 'BaseException' caught GeneratorExit
Handler 'BaseException' caught OSError
Handler 'BaseException' caught SystemExit
Handler 'BaseException' caught SystemError
Handler 'BaseException' caught KeyboardInterrupt
Handler 'BaseException' caught MemoryError
Handler 'BaseException' caught BaseException

Finally, in order to make this Python 2/3 agnostic, you can try and import exceptions and if that fails (which it does in Python 3), fall-back to importing builtins which contains all Exceptions; we search the dictionary by name so it makes no difference:

try:
    import exceptions
    excDict = vars(exceptions)
except ImportError:
    import builtins 
    excDict = vars(builtins)

I don't know if there's a smarter way to actually do this, another solution might be instead of having a try-except with a signle except, having 2 handlers, one for the black-listed exceptions and the other for the general case:

try:
    # calls that raise exceptions:
except tuple(blackSet) as be:  # Must go first, of course.
    raise be
except Exception as e:
    # handle the rest

Answer 2

The top-most exception is BaseException. There are two groups under that:

• Exception derived
• everything else

Things like Stopiteration, ValueError, TypeError, etc., are all examples of Exception.

Things like GeneratorExit, SystemExit and KeyboardInterrupt are not descended from Execption.

So the first step is to catch Exception and not BaseException which will allow you to easily terminate the program. I recommend also catching GeneratorExit as 1) it should never actually be seen unless it is raised manually; 2) you can log it and restart the loop; and 3) it is intended to signal a generator has exited and can be cleaned up, not that the program should exit.

The next step is to log each exception with enough detail that you have the possibility of figuring out what went wrong (when you later get around to debugging).

Finally, you have to decide for yourself which, if any, of the Exception derived exceptions you want to terminate on: I would suggest RuntimeError and MemoryError, although you may be able to get around those by simply stopping and restarting your server loop.

So, really, it's up to you.

If there is some other error (such as IOError when trying to load a config file) that is serious enough to quit on, then the code responsible for loading the config file should be smart enough to catch that IOError and raise SystemExit instead.

As far as whitelist/blacklist -- use a black list, as there should only be a handful, if any, Exception-based exceptions that you need to actually terminate the server on.