In Python, When yield cost over the return an list?

Question

In many case, people all ways say "use the yield to lazily create element." but I think everything have cost, include the yield and its iterator.

In effective nord eyes, I think it's nice question. so,for example, when I get an function.

def list_gen(n):
    if n > MAGIC_NUM:
        return xrange(n)
    else:
        return range(n)

How much dose the MAGIC_NUM is?

UPDATE sorry for this mistake, I'm origin meaning is compare the iterator's cost and list cost.

UPDATE AGAIN Please imaging an case. Whether have an condition, that the memory so limit that it's can't create an iterator.

ha, this question is more funny now.
UPDATE AGAIN Why does create an iterator and save the yield context are less then create a list? or How much does iterator cost ?(sorry for my insult) How many bytes?

Answer 1

You're mixing several things up.

def list_gen(n):
    i=0
    while i<n:
        yield i
        i += 1

This function is a generator. Calling it returns a generator object, which is an iterator.

An iterator is a thing that has next(), i.e. it can be traversed over once. An iterator is created over something using iter whenever you do a for i in something.

def list_gen(n):
    return range(n)

def list_gen(n):
    return xrange(n)

These functions are regular functions. One returns a list and the other returns an xrange object. Both lists and xranges are iterable, i.e. multiple independent iterators can be created for them.

---

So back to your question: You're asking whether to return a list or an xrange object.

That depends, obviously! It depends on what you want to do with the result.

• If you want to mutate it somehow, then you need a real list. Use range directly.

• If you only want to iterate over it, then it doesn't make a difference semantically: both an xrange object and a list returned by range will produce an iterator which iterates over the same sequence.

  However, if you use xrange, you'll never create the whole list in memory. Why create a full-fledged list object in memory if all you want to do is a simple iteration? You don't need to allocate a temporary large memory buffer whenever you want a for loop, right?

Hence: It's safe to stick with xrange, since the caller can always make a list out of it.

---

Let's confirm that with a benchmark. We want to know if it's faster to iterate over xranges than over lists constructed by range (including the cost of range call, of course).

Code:

import timeit

ns = [1,2,3, 5, 10, 50, 100]
print 'n', '\t', 'range', '\t', 'xrange'
for n in ns:
    t1 = timeit.timeit("for i in range({}): pass".format(n))
    t2 = timeit.timeit("for i in xrange({}): pass".format(n))
    print n, '\t', t1, '\t', t2

Result:

n       range           xrange
1       0.566222990493  0.418698436395
2       0.594136874362  0.477882061758
3       0.630704800817  0.488603362929
5       0.725149288913  0.540597548519
10      0.90297752809   0.687031507818
50      2.44493085566   1.89102105759
100     4.31189321914   3.33713522433

Answer 2

It has nothing to do with the length of the iterator you are generating, but with how you need to use it afterwards. If you only need to use it once then you should definitely go for yield, if you'll go on an use it multiple times you can skip yield and just get a regular list. Keep in mind generators you get using yield can only be iterated once.