How to catch an exception in the for loop iterator

Question

This is a for loop in Python:

for_stmt ::=  "for" target_list "in" expression_list ":" suite 

Normally, when yielding a value from the expression_list raises an exception, the loop aborts. Is there an elegant way (short of rewriting the loop using while True or something similar) to catch this exception and continue the loop?

Here is an example:

import csv  csv.field_size_limit(10)  reader = csv.reader(open('test.csv', 'r')) for line in reader:     print(line) 

with this file:

foo,bar,baz xxx,veryverylong,yyy abc,def,ghi 

This aborts at the second line. I would like a way to skip or log the failing lines and continue.

Answer 1

If your inner iterable can be continued after an exception, all you need to wrap it is a trivial generator:

def wrapper(gen):   while True:     try:       yield next(gen)     except StopIteration:       break     except Exception as e:       print(e) # or whatever kind of logging you want 

For example:

In [9]: list(wrapper(csv.reader(open('test.csv', 'r')))) field larger than field limit (10) Out[9]: [['foo', 'bar', 'baz'], ['abc', 'def', 'ghi']] 

On the other hand, if the inner iterator can't be continued after an exception, there's no way you can wrap it:

def raisinggenfunc():     yield 1     raise ValueError("spurious error")     yield 3  In [11]: list(wrapper(raisinggenfunc())) spurious error Out[11]: [1] 

Any generator created by calling a Python generator function or evaluating a generator expression will not be resumable.

In such a case, you need to find some way to create a new iterator that resumes iteration. For something like csv.reader, that would mean reading n lines from the file before wrapping it in a csv.reader. In other cases it might mean passing n to the constructor. In other cases—as with raisinggenfunc above, it's just not possible.

Answer 2

You could wrap the reader in another iterator that then handles the exceptions however you please.

class ExceptionHandlingIterator(object):     def __init__(self, iterable):         self._iter = iter(iterable)         self.handlers = []     def __iter__(self):         return self     def next(self):         try:             return self._iter.next()         except StopIteration as e:             raise e         except Exception as e:             for handler in self.handlers:                 handler(e)             return self.next()  csv_reader = ExceptionHandlingIterator(csv.reader(open('test.csv', 'r')) # attach handlers to the reader here for line in csv_reader:     print line