Appending a numpy array to a list - strange happenings

Question

Using Python3.5.3 in Spyder 3.1.3 on Raspbian on a Raspberry Pi. Appending two numpy-arrays to a list named 'list0' works well with allocated numpy arrays 'a' like:

import numpy as np

list0 = []
a = np.array([[1,2,3],[2,3,4]])
list0.append(a)
a = np.array([[11,12,13],[12,13,14]])
list0.append(a)

print("list0 =",list0)

works well, giving as output (a bit better formatted for the post):

list0 = [ array([[ 1,  2,  3], [ 2,  3,  4]]), 
          array([[11, 12, 13], [12, 13, 14]]) ]

Replacing the assignment to a using a loop, weird things happen:

import numpy as np
a = np.empty((3), int)
list0 = []
for idx in range(4):    
    for i in range(3):
        a[i] = idx*10 + i
    print("idx =",idx,"; a =",a)
    list0.append(a)
print("list0 =",list0)

The second line tells Python the shape of the array used (in my original case it is a three-dimensional array). For verification the generated arrays named 'a' are printed out. Appending the newly filled arrays 'a' to 'list0' finally shows four times the last line.

idx = 0 ; a = [ 0  1  2]
idx = 1 ; a = [10 11 12]
idx = 2 ; a = [20 21 22]
idx = 3 ; a = [30 31 32]
list0 = [ array([30, 31, 32]), array([30, 31, 32]), 
          array([30, 31, 32]), array([30, 31, 32]) ] 

I suppose that 'list0' simply contains four times a pointer to the array 'a' which only exists in one instance / memory range.

So: How can I physically append (copy?) each of the different arrays 'a' to the list? Is it a python bug or is it simply my misunderstanding of something? Certainly I should think more pythonian ;c)

Thanks for help, Peter

Answer 1

The problem

You're appending the same array a to your list0 4 times. Arrays like a are mutable objects, which means, among other things, that when you assign values to them the underlying object changes. Since the array is present in your list 4 times, those changes (seem to) show up in 4 different places.

Solution

You can fix the code you have with one small change. Append a copy of the array to your list, instead of the array itself:

import numpy as np
a = np.empty((3), int)
list0 = []
for idx in range(4):    
    for i in range(3):
        a[i] = idx*10 + i
    print("idx =",idx,"; a =",a)
    list0.append(a.copy())
print("list0 =",list0)

Output:

idx = 0 ; a = [0 1 2]
idx = 1 ; a = [10 11 12]
idx = 2 ; a = [20 21 22]
idx = 3 ; a = [30 31 32]
list0 = [array([0, 1, 2]), array([10, 11, 12]), array([20, 21, 22]), array([30, 31, 32])]

Optimized solution

Python/Numpy offer many better ways (both in terms of using fewer lines of code and running faster) to initialize arrays. For a bunch of ranges like this, here is a reasonable approach:

list0 = [np.arange(n*10, n*10+3) for n in range(4)]
print(list0)

Output:

[array([0, 1, 2]), array([10, 11, 12]), array([20, 21, 22]), array([30, 31, 32])]

You might also consider just using a single 2D array in place of a list of arrays. One single array is often easier to work with than a heterogenous mix of arrays in a list. Here's how you do that:

arr0 = np.array([np.arange(n*10, n*10+3) for n in range(4)])
print(arr0)

Output:

[[ 0  1  2]
 [10 11 12]
 [20 21 22]
 [30 31 32]]

Answer 2

Just do this:

list_to_append.append(np_array.copy())

In a nutshell, numpy arrays or lists are mutable objects, which means that you when you assign a numpy array or list to a variable, what you are really assigning are references to memory locations aka pointers.

In your case, "a" is a pointer, so what you are really doing is appending to list0 an address to the memory location pointed by "a", and not the actual values pointed by the pointer. Thus it means that each new position of "list0", after appending, turns out to be the same address of memory: "a".

So, instead of:

list0.append(a)

You call the copy() method of "a" that creates a new memory location for the new values of "a" and returns it:

list0.append(a.copy())