Python: check if object exists in generator

Question

I had a bug which was caused by the changing result of checking "if x in generator"

def primes(upper_limit):
    for n in range(2, upper_limit):
        if all(n % i > 0 for i in range(2, n)):
            yield n

first_hundred_primes = primes(100)

print(5 in first_hundred_primes)
print(5 in first_hundred_primes)
print(5 in first_hundred_primes)
print(5 in first_hundred_primes)
print(5 in first_hundred_primes)

This gives the output:

True
False
False
False
False

I'm assuming one isn't meant to check if an object exists in a generator, but if this is the case why doesn't it throw some error, and why does this work?

>>> hundred_generator = range(1,100)
>>> 50 in hundred_generator
True
>>> 50 in hundred_generator
True
>>> 50 in hundred_generator
True

I usually turn the generator into a set before I'm checking if some object exists in it (to speed up checking) and this fixes the problem, but I would very much like to know what's going on here?

Answer 1

When you iterate over the elements of a generator, you consume them.

Try this:

len(list(first_hundred_primes)) > 0
=> True
len(list(first_hundred_primes)) > 0
=> False

I.e. you're done consuming the elements the first time you use in (which iterates over them), or at list all the elements up to 5, thus the generator will not generate 5 again after that. After the second time, it won't generate anything anymore.

Your options:

1. convert your generator to a list (or a set) before using it: first_hundred_primes = list(first_hundred_primes)
2. create a new generator each time: 5 in primes(100); 5 in primes(100); ...
3. use itertools.tee

EDIT:

As to your question about range: range isn't a generator.

In python2, it simply returns a list. No problem there.

In python3, it returns a special object which looks like a collection. It doesn't have to actually store all the numbers in the range, it simply implements the list-operations based on the rule defining the range. E.g. len is implemented as stop-start. Since it stands for a collection, and not a generator, you can iterate over it many time, without "consuming" the elements.