How can I elide a function wrapper from the traceback in Python-3?

Question

The issue

The Phantom Menace

Say i wrote a function decorator which takes the function, and wraps it in another function like so:

# File example-1.py
from functools import wraps

def decorator(func):
    # Do something
    @wraps(func)
    def wrapper(*args, **kwargs):
        # Do something
        return func(*args, **kwargs)
        # Do something
    # Do something
    return wrapper

Now lets suppose the function I'm decorating raises an exception:

@decorator
def foo():
    raise Exception('test')

The result of running foo() will print out the following traceback (In any Python version):

Traceback (most recent call last):
  File "./example-1.py", line 20, in <module>
    foo()
  File "./example-1.py", line 11, in wrapper
    return func(*args, **kwargs)
  File "./example-1.py", line 18, in foo
    raise Exception('test')
Exception: test

Attack of the Clones

OK, now i look at my traceback and i see it goes through the wrapper function. What if I wrapped the function multiple times(presumably with a slightly more sophisticated decorator object which receives arguments in its constructor)? What if I use this decorator often in my code(I use it for logging, or profiling, or whatever)?

Traceback (most recent call last):
  File "./example-1.py", line 20, in <module>
    foo()
  File "./example-1.py", line 11, in wrapper
    return func(*args, **kwargs)
  File "./example-1.py", line 11, in wrapper
    return func(*args, **kwargs)
  File "./example-1.py", line 11, in wrapper
    return func(*args, **kwargs)
  File "./example-1.py", line 11, in wrapper
    return func(*args, **kwargs)
  File "./example-1.py", line 18, in foo
    raise Exception('test')
Exception: test

I don't want it "polluting" my traceback when i know from the function definition that the wrapper is there, and i don't want it showing up multiple times when the code snippet it displays is the unhelpful return func(*args, **kwargs)

Python 2

Revenge of the Sith

In Python-2, as this answer to a different question points out, the following trick does the job:

# In file example-2.py

def decorator(func):
    # Do something
    @wraps(func)
    def wrapper(*args, **kwargs):
        # Do something
        info = None
        try:
            return func(*args, **kwargs)
        except:
            info = sys.exc_info()
            raise info[0], info[1], info[2].tb_next
        finally:
            # Break the cyclical reference created by the traceback object
            del info
        # Do something
    # Do something
    return wrapper

By directly wrapping the call to the wrapped function with this idiom in the same block as the function I want to elide from the traceback, I effectively remove the current layer from the traceback and let the exception keep propagating. Every time the stack unwinding goes through this function, it removes itself from the traceback so this solution works perfectly:

Traceback (most recent call last):
  File "./example-2.py", line 28, in <module>
    foo()
  File "./example-2.py", line 26, in foo
    raise Exception('test')
Exception: test

(Note however that you can not encapsulate this idiom in another function, since as soon the stack will unwind from that function back into wrapper, it will still be added to the traceback)

Python 3

A New Hope

Now that we have this covered, lets move along to Python-3. Python-3 introduced this new syntax:

raise_stmt ::=  "raise" [expression ["from" expression]]

which allows chaining exceptions using the __cause__ attribute of the new exception. This feature is uninteresting to us, since it modifies the exception, not the traceback. Our goal is to be a completely transparent wrapper, as far as visibility goes, so this won't do.

Alternatively, we can try the following syntax, which promises to do what we want (code example taken from the python documentation):

raise Exception("foo occurred").with_traceback(tracebackobj)

Using this syntax we may try something like this:

# In file example-3
def decorator(func):
    # Do something
    @wraps(func)
    def wrapper(*args, **kwargs):
        # Do something
        info = None
        try:
            return func(*args, **kwargs)
        except:
            info = sys.exc_info()
            raise info[1].with_traceback(info[2].tb_next)
        finally:
            # Break the cyclical reference created by the traceback object
            del info
        # Do something
    # Do something
    return wrapper

The Empire Strikes Back

But, unfortunately, this does not do what we want:

Traceback (most recent call last):
  File "./example-3.py", line 29, in <module>
    foo()
  File "./example-3.py", line 17, in wrapper
    raise info[1].with_traceback(info[2].tb_next)
  File "./example-3.py", line 27, in foo
    raise Exception('test')
Exception: test

As you can see, the line executing the raise statement shows up in the traceback. This seems to come from the fact that while the Python-2 syntax sets the traceback from the third argument to raise as the function is being unwound, and thus it is not added to the traceback chain(as explained in the docs under Data Model), the Python-3 syntax on the other hand changes the traceback on the Exception object as an expression inside the functions context, and then passes it to the raise statement which adds the new location in code to the traceback chain (the explanation of this is very similar in Python-3).

A workaround that comes to mind is avoiding the "raise" [ expression ] form of the statement, and instead use the clean raise statement to let the exception propagate as usual but modify the exception objects __traceback__ attribute manually:

# File example-4
def decorator(func):
    # Do something
    @wraps(func)
    def wrapper(*args, **kwargs):
        # Do something
        info = None
        try:
            return func(*args, **kwargs)
        except:
            info = sys.exc_info()
            info[1].__traceback__ = info[2].tb_next
            raise
        finally:
            # Break the cyclical reference created by the traceback object
            del info
        # Do something
    # Do something
    return wrapper

But this doesn't work at all!

Traceback (most recent call last):
  File "./example-4.py", line 30, in <module>
    foo()
  File "./example-4.py", line 14, in wrapper
    return func(*args, **kwargs)
  File "./example-4.py", line 28, in foo
    raise Exception('test')
Exception: test

Return of the Jedi(?)

So, what else can i do? It seems like using the "traditional" way of doing this just won't work because of the change in syntax, and I wouldn't want to start messing with the traceback printing mechanism (using the traceback module) at the project level. This is because it'll be hard if not impossible to implement in an extensible which won't be disruptive to any other package that tries to change the traceback, print the traceback in a custom format at the top level, or otherwise do anything else related to the issue.

Also, can someone explain why in fact the last technique fails completely?

(I tried these examples on python 2.6, 2.7, 3.4, 3.6)

EDIT: After thinking about it for a while, in my opinion the python 3 behavior makes more sense, to the point that the python 2 behavior almost looks like a design bug, but I still think that there should be a way to do this kinda stuff.

Answer 1

The simple answer is that you shouldn't do that. Hiding things from the traceback is dangerous. You may think you don't want to show that line because it's trivial or "just a wrapper", but in general you wouldn't write the wrapper function if it didn't do something. Next thing you know there is a bug in the wrapper function which is now unfindable because the wrapper function has erased itself from the traceback.

Just deal with the extra lines in the traceback, or, if you really want, override sys.excepthook and filter them out at the top level. If you're worried about someone else overriding sys.excepthook too, then wrap all your code in a top-level function that does the exception printing itself. It isn't and shouldn't be easy to hide levels from the traceback.