Ignore exception from generator

Question

By using os.walk() to traverse the folder like this:

for subdir, dirs, files in os.walk(path):
    do something...

There will throw a exception: UnicodeDecodeError, I want to ignore the exception and continue, and I tried this:

try:
    for subdir, dirs, files in os.walk(path):
        do something...
except Exception, e:
    logging.exception(e)
    continue   # this continue is illegal

as the comment says, the continue in exception part is a syntax error. Is there a way to ignore the exception and continue to traverse?

The exception is thrown from os.walk(), so put the try/except within for can not catch the exception. The os.walk() will return a python generator, How to catch the exception within it?

Answer 1

Update:

I originally thought that the error was being raised by the do something... code. Since it is actually being raised by os.walk, you will need to do something a little different:

walker = os.walk(path)
while True:
    try:
        subdir, dirs, files = next(walker)
    except UnicodeDecodeError as e:
        logging.exception(e)
        continue
    except StopIteration:
        break

    do something...

Basically, this is taking advantage of the fact that os.walk returns a generator object. This allows us to call next on it and thereby have control over the iteration during each step.

The subdir, dirs, files = next(walker) line attempts to advance the iteration. If a UnicodeDecodeError is raised, it is logged and we continue on to the next step. If a StopIteration exception is raised, it means that we are done walking the directory tree. So, we break the loop.

---

Since continue needs to be inside the loop, you will need to move the try/except block in there as well:

for subdir, dirs, files in os.walk(path):
    try:
        do something...
    except Exception, e:
        logging.exception(e)
        continue   # this continue is *not* illegal

Also, doing:

except Exception, e:

has been deprecated. You should be using the as keyword in place of ,:

except Exception as e:

While you're at it, you should replace the generic Exception with the specific UnicodeDecodeError:

except UnicodeDecodeError as e:

You should always try to catch the most specific exception that you can. Otherwise, you run the risk of accidentally catching an exception that you did not mean to handle.

Answer 2

I had a similar situation while iterating over links using Beautiful Soup. This is the code I wrote for that:

class suppressed_iterator:
    def __init__(self, wrapped_iter, skipped_exc = Exception):
        self.wrapped_iter = wrapped_iter
        self.skipped_exc  = skipped_exc

    def __next__(self):
        while True:
            try:
                return next(self.wrapped_iter)
            except StopIteration:
                raise
            except self.skipped_exc:
                pass

class suppressed_generator:
    def __init__(self, wrapped_obj, skipped_exc = Exception):
        self.wrapped_obj = wrapped_obj
        self.skipped_exc = skipped_exc

    def __iter__(self):
        return suppressed_iterator(iter(self.wrapped_obj), self.skipped_exc)

An example:

class IHateThirteen:
    ''' Throws exception while iterating on value 13 '''

    def __init__(self, iterable):
        self.it = iter(iterable)

    def __iter__(self):
        return self

    def __next__(self):
        v = next(self.it)
        if v == 13:
            raise ValueError('I hate 13!')
        return v

# Outputs [10, 11, 12, 14, 15]
exception_at_thirteen = IHateThirteen([10, 11, 12, 13, 14, 15])
print(list(suppressed_generator(exception_at_thirteen)))

# Raises ValueError
exception_at_thirteen = IHateThirteen([10, 11, 12, 13, 14, 15])
print(list(exception_at_thirteen))

You can fix your code by using above code:

for subdir, dirs, files in suppressed_generator(os.walk(path)):
    do something...

Above code can be extended more to have callbacks per skipped exception types if needed, but it may be more pythonic to use iCodez's answer in such a case.