What happens when you try to delete a list element while iterating over it

Question

I'm iterating a list as follows:

some_list = [1, 2, 3, 4]
another_list = [1, 2, 3, 4]

for idx, item in enumerate(some_list):
    del some_list[idx]

for item in another_list:
    another_list.remove(item)

When I print out the contents of the lists

>>> some_list
[2, 4]
>>> another_list
[2, 4]

I'm aware that Python doesn't support modifying a list while iterating over it and the right way is to iterate over copy of list instead. But I want to know what exactly happens behind the scenes i.e. Why is the output of the above snippet [2, 4]?

Answer 1

You can use a self-made iterator that shows (in this case prints) the state of the iterator:

class CustomIterator(object):
    def __init__(self, seq):
        self.seq = seq
        self.idx = 0

    def __iter__(self):
        return self

    def __next__(self):
        print('give next element:', self.idx)
        for idx, item in enumerate(self.seq):
            if idx == self.idx:
                print(idx, '--->', item)
            else:
                print(idx, '    ', item)
        try:
            nxtitem = self.seq[self.idx]
        except IndexError:
            raise StopIteration
        self.idx += 1
        return nxtitem

    next = __next__  # py2 compat

Then use it around the list you want to check:

some_list = [1, 2, 3, 4]

for idx, item in enumerate(CustomIterator(some_list)):
    del some_list[idx]

This should illustrate what happens in that case:

give next element: 0
0 ---> 1
1      2
2      3
3      4
give next element: 1
0      2
1 ---> 3
2      4
give next element: 2
0      2
1      4

It only works for sequences though. It's more complicated for mappings or sets.

Answer 2

I want to know what exactly happens behind the scenes

As we know, every item in a list lives at its own unique index; which are in order, starting from 0. If we remove an item, any item with an index greater than the one we've removed has now been shifted down.

And here's why that matters:

foo = ['a', 'b', 'c', 'd']
for index in range(len(foo)):
    del foo[index]

In this loop, we're removing all the elements, so we should end up with foo == [], right? This is not the case. In our first trip through the loop, we remove the item at index 0, and the item at index 1 becomes the item at index 0. Our next time through the loop, we remove the item at index 1 which was previously the item at index 2.

In just the first two iterations, we've removed 'a' and 'c' from the array, *but we've neglected to remove 'b'. Once we get to the third iteration (where we though we'd remove index 2), there is no longer an element at index 2; only indices 0 and 1. An exception is raised when we try to remove the non-existent item at index 2, and the loop is stopped. The result is a mangled array that looks like this: ['a', 'd'].