Listing a filter object twice will return a blank list?

Question

I'm using the filter function in Python (3.3). I tried to convert the filter object to list. That's what I found:

>>> a=['1', '2', '3', None]
>>> b=filter(None,a)
>>> list(b)
['1', '2', '3']
>>> list(b)
[]

That's quite strange to me. Can anyone explain this?

Answer 1

In Python 3, filter() returns an iterator type, and like all iterators, can only be iterated over just once. The filter() iterator filters values on demand, it doesn't hold any filtered values in memory itself.

You can do the same with the list iterator, returned by iter():

>>> a = [1, 2, 3]
>>> b = iter(a)
>>> list(b)
[1, 2, 3]
>>> list(b)
[]

This happens because the .__next__() method of iterators are expected to raise StopIteration once exhausted, then must always raise StopIteration from there on out:

Once an iterator’s __next__() method raises StopIteration, it must continue to do so on subsequent calls. Implementations that do not obey this property are deemed broken.

filter() does this correctly:

>>> a = [1, 2, 3, None]
>>> b = filter(None, a)
>>> list(b)
[1, 2, 3]
>>> next(b)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

Here the next() function calls the .__next__() iterator method, and propagates the exception raised; list() on the other hand iterates until StopIteration, catches that exception and returns a list of whatever elements it managed to receive.

For completeness sake, in Python 2, filter() (as well as many other built-in functions and methods) returns a list, often wasting memory and cycles on building an intermediary list object that was then discarded again after iteration. By returning an iterator instead, the choice to materialize a list is up to the programmer instead.