What is the role of [:] in overwriting a list in a for loop?

Question

I came across a weird syntactical approach at work today that I couldn't wrap my head around. Let's say I have the following list:

my_list = [[1, 2, 3], [4, 5, 6]]

My objective is to filter each nested list according to some criteria and overwrite the elements of the list in place. So, let's say I want to remove odd numbers from each nested list such that my_list contains lists of even numbers, where the end result would look like this:

[[2], [4, 6]]

If I try to do this using a simple assignment operator, it doesn't work.

my_list = [[1, 2, 3], [4, 5, 6]]
for l in my_list:
    l = [num for num in l if num % 2 == 0]
print(my_list)

Output: [[1, 2, 3], [4, 5, 6]]

However, if I "slice" the list, it provides the expected output.

my_list = [[1, 2, 3], [4, 5, 6]]
for l in my_list:
    l[:] = [num for num in l if num % 2 == 0]
print(my_list)

Output: [[2], [4, 6]]

My original hypothesis was that l was a newly created object that didn't actually point to the corresponding object in the list, but comparing the outputs of id(x[i]), id(l), and id(l[:]) (where i is the index of l in x), I realized that l[:] was the one with the differing id. So, if Python is creating a new object when I assign to l[:] then how does Python know to overwrite the existing object of l? Why does this work? And why doesn't the simple assignment operator l = ... work?

Answer 1

It's subtle.

Snippet one:

my_list = [[1, 2, 3], [4, 5, 6]]
    for l in my_list:
        l = [num for num in l if num % 2 == 0]

Why doesn't this work? Because when you do l = , you're only reassigning the variable l, not making any change to its value.

If we write the loop out "manually", it hopefully will become more clear why this strategy fails:

my_list = [[1, 2, 3], [4, 5, 6]]

# iteration 1
l = my_list[0]
l = [num for num in l if num % 2 == 0]

# iteration 2
l = my_list[1]
l = [num for num in l if num % 2 == 0]

Snippet two:

my_list = [[1, 2, 3], [4, 5, 6]]
for l in my_list:
    l[:] = [num for num in l if num % 2 == 0]

Why does this work? Because by using l[:] = , you're actually modifying the value that l references, not just the variable l. Let me elaborate.

Generally speaking, using [:] notation (slice notation) on lists allows one to work with a section of the list.

The simplest use is for getting values out of a list; we can write a[n:k] to get the nth, item n+1st item, etc, up to k-1. For instance:

>>> a = ["a", "very", "fancy", "list"]
>>> print(a[1:3])
['very', 'fancy']

Python also allows use of slice notation on the left-side of a =. In this case, it interprets the notation to mean that we want to update only part of a list. For instance, we can replace "very", "fancy" with "not", "so", "fancy" like so:

>>> print(a)
['a', 'very', 'fancy', 'list']
>>> a[1:3] = ["not", "so", "fancy"]
>>> print(a)
['a', 'not', 'so', 'fancy', 'list']

When using slice syntax, Python also provides some convenient shorthand. Instead of writing [n:k], we can omit n or k or both.

If we omit n, then our slice looks like [:k], and Python understands it to mean "up to k", i.e., the same as [0:k].

If we omit k, then our slice looks like a[n:], and Python understands it to mean "n and after", i.e., the same as a[n:len(a)].

If we omit both, then both rules take place, so a[:] is the same as a[0:len(a)], which is a slice over the entire list.

Examples:

>>> print(a)
['a', 'not', 'so', 'fancy', 'list']
>>> print(a[2:4])
['so', 'fancy']
>>> print(a[:4])
['a', 'not', 'so', 'fancy']
>>> print(a[2:])
['so', 'fancy', 'list']
>>> print(a[:])
['a', 'not', 'so', 'fancy', 'list']

Crucially, this all still applies if we are using our slice on the left-hand side of a =:

>>> print(a)
['a', 'not', 'so', 'fancy', 'list']
>>> a[:4] = ["the", "fanciest"]
>>> print(a)
['the', 'fanciest', 'list']

And using [:] means to replace every item in the list:

>>> print(a)
['the', 'fanciest', 'list']
>>> a[:] = ["something", "completely", "different"]
>>> print(a)
['something', 'completely', 'different']

Okay, so far so good.

They key thing to note is that using slice notation on the left-hand side of a list updates the list in-place. In other words, when I do a[1:3] =, the variable a is never updated; the list that it references is.

We can see this with id(), as you were doing:

>>> print(a)
['something', 'completely', 'different']
>>> print(id(a))
139848671387072
>>> a[1:] = ["truly", "amazing"]
>>> print(a)
['something', 'truly', 'amazing']
>>> print(id(a))
139848671387072

Perhaps more pertinently, this means that if a were a reference to a list within some other object, then using a[:] = will update the list within that object. Like so:

>>> list_of_lists = [ [1, 2], [3, 4], [5, 6] ]
>>> second_list = list_of_lists[1]
>>> print(second_list)
[3, 4]
>>> second_list[1:] = [2, 1, 'boom!']
>>> print(second_list)
[3, 2, 1, 'boom!']
>>> print(list_of_lists)
[[1, 2], [3, 2, 1, 'boom!'], [5, 6]]