Speed up python loop

Question

I'm trying to speed up the following python code:

for j in range(4,len(var_s),3):
    mag_list.append(float(var_s[j]))
mag_list = [value for value in mag_list if value != 99.]
med_mag = np.median(mag_list)

Is there a nice way to combine the two for-loops into one? This way, it is really slow. What I need is to extract every third entry from the var_s list, beginning with the fifths, if the value of that entry is not equal to 99. Of the resulting list, I need the median. Thanks!

Answer 1

You could probably try:

mag_list = [value for value in var_s[4::3] if value != 99.]

depending on var_s, you might do better using itertools.islice(var_s,4,None,3), but that would definitely need to be timed to know.

Perhaps you'd do even better if you stuck with numpy the whole way:

vs = np.array(var_s[4::3],dtype=np.float64)  #could slice after array conversion too ...
med_mag = np.median(vs[vs!=99.])

Again, this would need to be timed to see how it performed relative to the others.

Answer 2

mag_list = filter(lambda x: x != 99, var_s[4::3])

Ok so, here are some timeit trials, all in Python 2.7.2:

The setup:

>>> from random import seed, random
>>> from timeit import Timer
>>> from itertools import islice, ifilter, imap
>>> seed(1234); var_s = [random() for _ in range(100)]

Using a for loop:

>>> def using_for_loop():
...     mag_list = []
...     for j in xrange(4, len(var_s), 3):
...             value = float(var_s[j])
...             if value != 99: mag_list.append(value)
... 
>>> Timer(using_for_loop).timeit()
11.596584796905518

Using map and filter:

>>> def using_map_filter():
...     map(float, filter(lambda x: x != 99, var_s[4::3]))
... 
>>> Timer(using_map_filter).timeit()
8.643505096435547

Using islice, imap, ifilter:

>>> def using_itertools():
...     list(imap(float, ifilter(lambda x: x != 99, islice(var_s, 4, None, 3)))) 
... 
>>> Timer(using_itertools).timeit()
11.311019897460938

Using a list comprehension and islice:

>>> def using_list_comp():
...     [float(v) for v in islice(var_s, 4, None, 3) if v != 99]
... 
>>> Timer(using_list_comp).timeit()
8.52650499343872
>>> 

In conclusion, using a list comprehension with islice is the fastest, followed by the only slightly slower use of map and filter.