Previous error being masked by current exception context

Question

The following is an example I found at the website for Doug Hellman in a file named "masking_exceptions_catch.py". I can't locate the link at the moment. The exception raised in throws() is discarded while that raised by cleanup() is reported.

In his article, Doug remarks that the handling is non-intuitive. Halfway expecting it to be a bug or limitation in the Python version at the time it was written (circa 2009), I ran it in the current production release of Python for the Mac (2.7.6). It still reports the exception from cleanup(). I find this somewhat amazing and would like to see a description of how it is actually correct or desirable behavior.

#!/usr/bin/env python

import sys
import traceback

def throws():
    raise RuntimeError('error from throws')

def nested():
    try:
        throws()
    except:
        try:
            cleanup()
        except:
            pass # ignore errors in cleanup
        raise # we want to re-raise the original error

def cleanup():
    raise RuntimeError('error from cleanup')

def main():
    try:
        nested()
        return 0
    except Exception, err:
        traceback.print_exc()
        return 1

if __name__ == '__main__':
    sys.exit(main())

Program output:

$ python masking_exceptions_catch.py
Traceback (most recent call last):
  File "masking_exceptions_catch.py", line 24, in main
    nested()
  File "masking_exceptions_catch.py", line 14, in nested
    cleanup()
  File "masking_exceptions_catch.py", line 20, in cleanup
    raise RuntimeError('error from cleanup')
RuntimeError: error from cleanup

Answer 1

Circling back around to answer. I'll start by not answering your question. :-)

Does this really work?

def f():
    try:
        raise Exception('bananas!')
    except:
        pass
    raise

So, what does the above do? Cue Jeopardy music.

---

Alright then, pencils down.

# python 3.3
      4     except:
      5         pass
----> 6     raise
      7 

RuntimeError: No active exception to reraise

# python 2.7
      1 def f():
      2     try:
----> 3         raise Exception('bananas!')
      4     except:
      5         pass

Exception: bananas!

Well, that was fruitful. For fun, let's try naming the exception.

def f():
    try:
        raise Exception('bananas!')
    except Exception as e:
        pass
    raise e

What now?

# python 3.3
      4     except Exception as e:
      5         pass
----> 6     raise e
      7 

UnboundLocalError: local variable 'e' referenced before assignment

# python 2.7
      4     except Exception as e:
      5         pass
----> 6     raise e
      7 

Exception: bananas!

Exception semantics changed pretty drastically between python 2 and 3. But if python 2's behavior is at all surprising to you here, consider: it's basically in line with what python does everywhere else.

try:
    1/0
except Exception as e: 
    x=4
#can I access `x` here after the exception block?  How about `e`?

try and except are not scopes. Few things are, actually, in python; we have the "LEGB Rule" to remember the four namespaces - Local, Enclosing, Global, Builtin. Other blocks simply aren't scopes; I can happily declare x within a for loop and expect to still be able to reference it after that loop.

So, awkward. Should exceptions be special-cased to be confined to their enclosing lexical block? Python 2 says no, python 3 says yes. But I'm oversimplifying things here; bare raise is what you initially asked about, and the issues are closely related but not actually the same. Python 3 could have mandated that named exceptions are scoped to their block without addressing the bare raise thing.

What does bare raise do‽

Common usage is to use bare raise as a means to preserve the stack trace. Catch, do logging/cleanup, reraise. Cool, my cleanup code doesn't appear in the traceback, works 99.9% of the time. But things can go south when we try to handle nested exceptions within an exception handler. Sometimes. (see examples at the bottom for when it is/isn't a problem)

Intuitively, no-argument raise would properly handle nested exception handlers, and figure out the correct "current" exception to reraise. That's not exactly reality, though. Turns out that - getting into implementation details here - exception info is saved as a member of the current frame object. And in python 2, there's simply no plumbing to handle pushing/popping exception handlers on a stack within a single frame; there's just simply a field that contains the last exception, irrespective of any handling we may have done to it. That's what bare raise grabs.

6.9. The raise statement

raise_stmt ::= "raise" [expression ["," expression ["," expression]]]

If no expressions are present, raise re-raises the last exception that was active in the current scope.

So, yes, this is a problem deep within python 2 related to how traceback information is stored - in Highlander tradition, there can be only one (traceback object saved to a given stack frame). As a consequence, bare raise reraises what the current frame believes is the "last" exception, which isn't necessarily the one that our human brains believe is the one specific to the lexically-nested exception block we're in at the time. Bah, scopes!

So, fixed in python 3?

Yes. How? New bytecode instruction (two, actually, there's another implicit one at the start of except handlers) but really who cares - it all "just works" intuitively. Instead of getting RuntimeError: error from cleanup, your example code raises RuntimeError: error from throws as expected.

I can't give you an official reason why this was not included in python 2. The issue has been known since PEP 344, which mentions Raymond Hettinger raising the issue in 2003. If I had to guess, fixing this is a breaking change (among other things, it affects the semantics of sys.exc_info), and that's often a good enough reason not to do it in a minor release.

Options if you're on python 2:

1) Name the exception you intend to reraise, and just deal with a line or two being added to the bottom of your stack trace. Your example nested function becomes:

def nested():
    try:
        throws()
    except BaseException as e:
        try:
            cleanup()
        except:
            pass 
        raise e

And associated traceback:

Traceback (most recent call last):
  File "example", line 24, in main
    nested()
  File "example", line 17, in nested
    raise e
RuntimeError: error from throws

So, the traceback is altered, but it works.

1.5) Use the 3-argument version of raise. A lot of people don't know about this one, and it is a legitimate (if clunky) way to preserve your stack trace.

def nested():
    try:
        throws()
    except:
        e = sys.exc_info()
        try:
            cleanup()
        except:
            pass 
        raise e[0],e[1],e[2]

sys.exc_info gives us a 3-tuple containing (type, value, traceback), which is exactly what the 3-argument version of raise takes. Note that this 3-arg syntax only works in python 2.

2) Refactor your cleanup code such that it cannot possibly throw an unhandled exception. Remember, it's all about scopes - move that try/except out of nested and into its own function.

def nested():
    try:
        throws()
    except:
        cleanup()
        raise

def cleanup():
    try:
        cleanup_code_that_totally_could_raise_an_exception()
    except:
        pass

def cleanup_code_that_totally_could_raise_an_exception():
    raise RuntimeError('error from cleanup')

Now you don't have to worry; since the exception never made it to nested's scope, it won't interfere with the exception you intended to reraise.

3) Use bare raise like you were doing before you read all this and live with it; cleanup code doesn't usually raise exceptions, right? :-)