python function call with/without list comprehension [duplicate]

Question

I have below two functions:

def foo(n=50000):
    return sum(i*i for i in range(n))  # just called sum() directly without 

def bar(n=50000):
    return sum([i*i for i in range(n)])  # passed constructed list to sum()

I was hoping that foo will run faster then bar but I have checked in ipython with %%timeit that foo is taking slightly longer then bar

In [2]: %%timeit
   ...: foo(50000)
   ...: 
100 loops, best of 3: 4.22 ms per loop

In [3]: %%timeit
   ...: bar(50000)
   ...: 
100 loops, best of 3: 3.45 ms per loop
In [4]: %%timeit
   ...: foo(10000000)
   ...: 
1 loops, best of 3: 1.02 s per loop

In [5]: %%timeit
   ...: bar(10000000)
   ...: 
1 loops, best of 3: 869 ms per loop

The difference increases as I increase value of n hence I tried to check function with dis.dis(foo) and dis.dis(bar) but it was identical.

So what would be the cause of such time difference between both methods?

Answer 1

There are plenty of great answers about generators, so I won't elaborate on that.

Generators keep state. They are slightly slower if you do very fast operations (like using sum, but if you use an I/O command there won't be much difference). The upside for generators is that they don't load all the items to memory in advance, where lists does.

This is what happens when you iterate a list (in very high-level):

• You load all items of the list to memory
• Asking for the next element just gives you the pointer to that object

Compare that to a generator:

• You don't have all the items on memory. Just one item at a time.
• Asking for the next element resumes the generator object, running the code until it reaches the yield statement.
• Then it yields the object's address in memory so you can access it.

This extra step in the middle, is the diff in your tests.

So, generators are used commonly where you deal with huge amount of data that needs to be loaded on to memory. (There are more use-cases for generators ofcourse, like coroutines)

Do an expirement with huge files and a for loop printing the lines. At some point you will get out of memory exception when using lists. Then try using generators, they won't go out of memory..