Exception handling when errors may occur in main program or in cleanup

Question

This is with Python 2.6.6 (default) on Debian Squeeze. Consider the following Python code.

import sys
try:
    raise Exception("error in main")
    pass
except:
    exc_info = sys.exc_info()
finally:
    try:
        print "cleanup - always run"
        raise Exception("error in cleanup")
    except:
        import traceback
        print >> sys.stderr, "Error in cleanup"
        traceback.print_exc()
    if 'exc_info' in locals():
        raise exc_info[0], exc_info[1], exc_info[2]

print "exited normally"

The error obtained is

Error in cleanup
Traceback (most recent call last):
  File "<stdin>", line 10, in <module>
Exception: error in cleanup
cleanup - always run
Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
Exception: error in main

The idea is to cope with a situation where either some code or the cleanup of that code (which is always run) or both, gives an error. There is some discussion of this, for example, by Ian Bicking in Re-raising Exceptions. At the end of that post, (see Update:) he describes how to handle the similar case of code + rollback/revert (only run in case of error).

I fiddled with this and came up with the code above, which is a bit of a monstrosity. In particular, if there is only an error in cleanup (commenting out raise Exception("error in main")), the code still exits normally, though it does print out a traceback. Currently, I'm giving the non-cleanup error priority, so it gets to stop the program.

Ideally I'd like either error to stop the program, but that doesn't seem to easy to arrange. Python seems to only want to raise one error, losing the others if any, and by default it is usually the last one. Rearranging this gives rise to convolutions like above.

Also the use of locals() is a bit ugly. Can one do better?

EDIT: srgerg's answer introduced me to the notion of the context managers and the with keyword. In addition to PEP 343, the other relevant bits of documentation I found are (in no particular order). Context Manager Types, The with statement, and http://docs.python.org/reference/datamodel.html#context-managers. This certainly seems like a big improvement on previous approaches to this, i.e. spaghetti code involving trys, excepts, and finallys.

To summarize, there are two things that I want such a solution to give me.

1. The ability for an exception in either the main code or in the cleanup to stop the program in its tracks. Context managers do this, because if the body of the with loop has an exception and the body of the exit does not, then that exception is propagated. If exit throws an exception and the body of the with loop does not, then that is propagated. if both throw an exception, then the exit exception is propagated and the one from the body of the while loop is suppressed. This is all documented i.e. from Context Manager Types,

   contextmanager.exit(exc_type, exc_val, exc_tb)

   Exit the runtime context and return a Boolean flag indicating if any exception that occurred should be suppressed. [...] Returning a true value from this method will cause the with statement to suppress the exception and continue execution with the statement immediately following the with statement. Otherwise the exception continues propagating after this method has finished executing. Exceptions that occur during execution of this method will replace any exception that occurred in the body of the with
   statement. [...] The exception passed in should never be reraised explicitly. instead, this method should return a false value to indicate that the method completed successfully and does not want to suppress the raised exception.

2. If there are exceptions in both places, I want to see tracebacks from both, even if technically only one exception is thrown. This is true based on experimentation, because if both throw an exception, then the exit exception is propagated, but the traceback from the body of the while loop is still printed, as in srgerg's answer. However, I can't find this documented anywhere, which is unsatisfactory.

Answer 1

Ideally, you'd use the python with statement to handle the cleanup within the try ... except block, which would look something like this:

class Something(object):
    def __enter__(self):
        print "Entering"

    def __exit__(self, t, v, tr):
        print "cleanup - always runs"
        raise Exception("Exception occurred during __exit__")

try:
    with Something() as something:
        raise Exception("Exception occurred!")
except Exception, e:
    print e
    import traceback
    traceback.print_exc(e)

print "Exited normally!"

When I run this, it prints:

Entering
cleanup - always runs
Exception occurred during __exit__
Traceback (most recent call last):
  File "s3.py", line 11, in <module>
    raise Exception("Exception occurred!")
  File "s3.py", line 7, in __exit__
    raise Exception("Exception occurred during __exit__")
Exception: Exception occurred during __exit__
Exited normally!

Note, either exception will terminate the program, and can be dealt with in the except statement.

Edit: According to the with statement documentation linked to above, the __exit__() method should only raise an exception if there is an error inside __exit__() - that is, it should not re-raise the exception passed into it.

This is a problem if both the code in the with statement and the __exit__() method raise an exception. In that case, the exception that is caught in the except clause is the one raised in __exit__(). If you want the one raised in the with statement, you can do something like this:

class Something(object):
    def __enter__(self):
        print "Entering"

    def __exit__(self, t, v, tr):
        print "cleanup - always runs"
        try:
            raise Exception("Exception occurred during __exit__")
        except Exception, e:
            if (t, v, tr) != (None, None, None):
                # __exit__ called with an existing exception
                return False
            else:
                # __exit__ called with NO existing exception
                raise

try:
    with Something() as something:
        raise Exception("Exception occurred!")
        pass
except Exception, e:
    print e
    traceback.print_exc(e)
    raise

print "Exited normally!"

This prints:

Entering
cleanup - always runs
Exception occurred!
Traceback (most recent call last):
  File "s2.py", line 22, in <module>
    raise Exception("Exception occurred!")
Exception: Exception occurred!
Traceback (most recent call last):
  File "s2.py", line 22, in <module>
   raise Exception("Exception occurred!")
Exception: Exception occurred!

Answer 2

A similar behaviour can be obtained by providing a customized exception hook:

import sys, traceback

def excepthook(*exc_info):
    print "cleanup - always run"
    raise Exception("error in cleanup")
    traceback.print_exception(*exc_info)
sys.excepthook = excepthook

raise Exception("error in main")

Example output:

cleanup - always run
Error in sys.excepthook:
Traceback (most recent call last):
  File "test.py", line 5, in excepthook
    raise Exception("error in cleanup")
Exception: error in cleanup

Original exception was:
Traceback (most recent call last):
  File "test.py", line 9, in <module>
    raise Exception("error in main")
Exception: error in main

In this example the code works as follows:

• If an exception isn't caught, excepthook is executed.
• Before printing the exception, excepthook runs some cleanup code (that was under finally in the original question).
• If an exception is raised in the hook, that exception is printed and, after that, the original exception is also printed.

Note: I haven't found any documentation regarding the printing of the original exception when something fails in the hook, but I've seen this behaviour in both cpython and jython. In particular, in cpython I've seen the following implementation:

void
PyErr_PrintEx(int set_sys_last_vars)
{
    ...
    hook = PySys_GetObject("excepthook");
    if (hook) {
        ...
        if (result == NULL) {
            ...
            PySys_WriteStderr("Error in sys.excepthook:\n");
            PyErr_Display(exception2, v2, tb2);
            PySys_WriteStderr("\nOriginal exception was:\n");
            PyErr_Display(exception, v, tb);
            ...
        }
    }
}