How to limit python traceback to specific files

Question

I write a lot of Python code that uses external libraries. Frequently I will write a bug, and when I run the code I get a big long traceback in the Python console. 99.999999% of the time it's due to a coding error in my code, not because of a bug in the package. But the traceback goes all the way to the line of error in the package code, and either it takes a lot of scrolling through the traceback to find the code I wrote, or the traceback is so deep into the package that my own code doesn't even appear in the traceback.

Is there a way to "black-box" the package code, or somehow only show traceback lines from my code? I'd like the ability to specify to the system which directories or files I want to see traceback from.

Answer 1

In order to print your own stacktrace, you would need to handle all unhandled exceptions yourself; this is how the sys.excepthook becomes handy.

The signature for this function is sys.excepthook(type, value, traceback) and its job is:

This function prints out a given traceback and exception to sys.stderr.

So as long as you can play with the traceback and only extract the portion you care about you should be fine. Testing frameworks do that very frequently; they have custom assert functions which usually does not appear in the traceback, in other words they skip the frames that belong to the test framework. Also, in those cases, the tests usually are started by the test framework as well.

You end up with a traceback that looks like this:

[ custom assert code ] + ... [ code under test ] ... + [ test runner code ]

How to identify your code.

You can add a global to your code:

__mycode = True 

Then to identify the frames:

def is_mycode(tb):   globals = tb.tb_frame.f_globals   return globals.has_key('__mycode') 

How to extract your frames.

1. skip the frames that don't matter to you (e.g. custom assert code)
2. identify how many frames are part of your code -> length

3. extract length frames

   def mycode_traceback_levels(tb):   length = 0   while tb and is_mycode(tb):     tb = tb.tb_next     length += 1   return length 

Example handler.

def handle_exception(type, value, tb):   # 1. skip custom assert code, e.g.   # while tb and is_custom_assert_code(tb):   #   tb = tb.tb_next   # 2. only display your code   length = mycode_traceback_levels(tb)   print ''.join(traceback.format_exception(type, value, tb, length)) 

install the handler:

sys.excepthook = handle_exception 

What next?

You could adjust length to add one or more levels if you still want some info about where the failure is outside of your own code.

see also https://gist.github.com/dnozay/b599a96dc2d8c69b84c6

Answer 2

As others suggested, you could use sys.excepthook:

This function prints out a given traceback and exception to sys.stderr.

When an exception is raised and uncaught, the interpreter calls sys.excepthook with three arguments, the exception class, exception instance, and a traceback object. In an interactive session this happens just before control is returned to the prompt; in a Python program this happens just before the program exits. The handling of such top-level exceptions can be customized by assigning another three-argument function to sys.excepthook.

(emphasis mine)

It's possible to filter a traceback extracted by extract_tb (or similar functions from the traceback module) based on specified directories.

Two functions that can help:

from os.path import join, abspath from traceback import extract_tb, format_list, format_exception_only  def spotlight(*show):     ''' Return a function to be set as new sys.excepthook.         It will SHOW traceback entries for files from these directories. '''     show = tuple(join(abspath(p), '') for p in show)      def _check_file(name):         return name and name.startswith(show)      def _print(type, value, tb):         show = (fs for fs in extract_tb(tb) if _check_file(fs.filename))         fmt = format_list(show) + format_exception_only(type, value)         print(''.join(fmt), end='', file=sys.stderr)      return _print  def shadow(*hide):     ''' Return a function to be set as new sys.excepthook.         It will HIDE traceback entries for files from these directories. '''     hide = tuple(join(abspath(p), '') for p in hide)      def _check_file(name):         return name and not name.startswith(hide)      def _print(type, value, tb):         show = (fs for fs in extract_tb(tb) if _check_file(fs.filename))         fmt = format_list(show) + format_exception_only(type, value)         print(''.join(fmt), end='', file=sys.stderr)      return _print 

They both use the traceback.extract_tb. It returns "a list of “pre-processed” stack trace entries extracted from the traceback object"; all of them are instances of traceback.FrameSummary (a named tuple). Each traceback.FrameSummary object has a filename field which stores the absolute path of the corresponding file. We check if it starts with any of the directory paths provided as separate function arguments to determine if we'll need to exclude the entry (or keep it).

---

Here's an Example:

The enum module from the standard library doesn't allow reusing keys,

import enum enum.Enum('Faulty', 'a a', module=__name__) 

yields

Traceback (most recent call last):   File "/home/vaultah/so/shadows/main.py", line 23, in <module>     enum.Enum('Faulty', 'a a', module=__name__)   File "/home/vaultah/cpython/Lib/enum.py", line 243, in __call__     return cls._create_(value, names, module=module, qualname=qualname, type=type, start=start)   File "/home/vaultah/cpython/Lib/enum.py", line 342, in _create_     classdict[member_name] = member_value   File "/home/vaultah/cpython/Lib/enum.py", line 72, in __setitem__     raise TypeError('Attempted to reuse key: %r' % key) TypeError: Attempted to reuse key: 'a' 

We can restrict stack trace entries to our code (in /home/vaultah/so/shadows/main.py).

import sys, enum sys.excepthook = spotlight('/home/vaultah/so/shadows') enum.Enum('Faulty', 'a a', module=__name__) 

and

import sys, enum sys.excepthook = shadow('/home/vaultah/cpython/Lib') enum.Enum('Faulty', 'a a', module=__name__) 

give the same result:

  File "/home/vaultah/so/shadows/main.py", line 22, in <module>     enum.Enum('Faulty', 'a a', module=__name__) TypeError: Attempted to reuse key: 'a' 

There's a way to exclude all site directories (where 3rd party packages are installed - see site.getsitepackages)

import sys, site, jinja2 sys.excepthook = shadow(*site.getsitepackages()) jinja2.Template('{%}') # jinja2.exceptions.TemplateSyntaxError: unexpected '}' #     Generates ~30 lines, but will only display 4 

Note: Don't forget to restore sys.excepthook from sys.__excepthook__. Unfortunately, you won't be able to "patch-restore" it using a context manager.