Python: Having difficulty filtering out strings containing certain substrings

Question

I have a list of strings that I'm trying to filter. If it contains the word blahblah, then I want to keep it. I tried it the first way, and I've gone over my code over and over, but I don't know why it retains some strings that don't have the word 'blahblah' in them. However, I tried it a second way, and it worked. I'm curious why the first way doesn't work.

The first method:

for item in my_list:
    if 'blahblah' not in item:
        my_list.remove(item)

The second method:

my_new_list = []
for m in my_list:
    if 'blahblah' in m:
        my_new_list.append(p)

The second method gives me what I want. I painstakingly inspected every element of the list generated from the second list.

1. Why is it that the first method doesn't work correctly?
2. If I go with the second method, will my code run any slower if I have an extremely large list?

Answer 1

To answer first question, its because when iterating over a list, internally Python keeps track of the index at which the current iteration has reached, when you remove the item from the list, it shifts the indexes of the elements, after the removed element, towards the left by 1 (decreases those indexes by 1) , hence in the next iteration you may end up skipping one element (because of this shifting).

---

To answer second question, it should be faster than removal method, since .remove() is an O(n) operation, where it needs to find the element to remove and then remove it, compared to that, .append() would be faster.

A bit faster method would be to use list comprehension -

my_new_list = [m for m in my_list if 'blahblah' in m]

If you want to change the my_list in place, you can use [:] on the left side of the assignment -

my_list[:] = [m for m in my_list if 'blahblah' in m]

Demo -

>>> my_list = ['blahblah','asdas']
>>> [m for m in my_list if 'blahblah' in m]
['blahblah']

Timing comparison between the two methods (as requested in the comments) -

In [4]: def func1():
   ...:     my_list = ['blahblah' for _ in range(100)]
   ...:     my_list[:] = [m for m in my_list if 'blahblah' in m]
   ...:

In [5]: def func2():
   ...:     my_list = ['blahblah' for _ in range(100)]
   ...:     new_list = [m for m in my_list if 'blahblah' in m]
   ...:

In [6]: %timeit func1()
100000 loops, best of 3: 13.9 µs per loop

In [7]: %timeit func2()
100000 loops, best of 3: 13.2 µs per loop

In [8]: %timeit func1()
100000 loops, best of 3: 13.9 µs per loop

In [9]: %timeit func2()
100000 loops, best of 3: 13.2 µs per loop

In [10]: %timeit func1()
100000 loops, best of 3: 13.8 µs per loop

In [11]: %timeit func2()
100000 loops, best of 3: 13.3 µs per loop