How to make python gracefully fail?

Question

I was just wondering how do you make python fail in a user defined way in all possible errors.

For example, I'm writing a program that processes a (large) list of items, and some of the items may not be in the format I defined. If python detects an error, it currently just spits out an ugly error message and stop the whole process. However, I want it to just outputs the error to somewhere together with some context and then move on to the next item.

If anyone can help me with this it would be greatly appreciated!

Thanks a lot!

Jason

Answer 1

The following are a few basic strategies I regularly use in my more-than-trivial scripts and medium-size applications.

Tip 1: Trap the error at every level where it makes sense to continue processing. In your case it may be in the inside the loop. You don't have to protect every single line or every single function call, but only the places where it makes a difference to survive the error.

Tip 2: Use the logging module to report what happened in a way that is configurable independently from how you compose the module with other modules in a larger applications. Start importing the root logger in your module, then, using it in a few different places, you may eventually figure out a more sensible logging hierarchy.

import logging
logger = logging.getLogger()

for item in items:
    try:
        process(item)
    except Exception, exc:
        logger.warn("error while processing item: %s", exc)

Tip 3: define an "application exception", eventually you may want to define a hierarchy of such exception but this is better discovered when the need arise. Use such exception(s) to "bubble out" when the data you are dealing with are not what you expected or to signal inconsistent situations, while separating them from the normal standard exception arising from regular bugs or problems outside the modeled domain (IO errors etc).

class DomainException(Exception):
    """Life is not what I expected"""

def process(item):
    # There is no way that this item can be processed, so bail out quickly.
    # Here you are assuming that your caller will report this error but probably
    # it will be able to process the other items.
    if item.foo > item.bar:
        raise DomainException("bad news")

    # Everybody knows that every item has more that 10 wickets, so
    # the following instruction is assumed always being successful.
    # But even if luck is not on our side, our caller will be able to
    # cope with this situation and keep on working
    item.wickets[10] *= 2

The main function is the outmost checkpoint: finally deal here with the possible ways your task finished. If this was not a shell script (but e.g. the processing beneath a dialog in an UI application or an operation after a POST in a web application) only the way you report the error changes (and the use of the logging method completely separates the implementation of the processing from its interface).

def main():
    try:
        do_all_the_processing()
        return 0
    except DomainException, exc:
        logger.error("I couldn't finish. The reason is: %s", exc)
        return 1
    except Exception, exc:
        logger.error("Unexpected error: %s - %s", exc.__class__.__name__, exc)
        # In this case you may want to forward a stacktrace to the developers via e-mail
        return 1
    except BaseException:
        logger.info("user stop") # this deals with a ctrl-c
        return 1

if __name__ == '__main__':
    sys.exit(main())