Create a dynamic 2D numpy array on the fly

Question

I am having a hard time creating a numpy 2D array on the fly.

So basically I have a for loop something like this.

for ele in huge_list_of_lists:
   instance = np.array(ele) 

creates a 1D numpy array of this list and now I want to append it to a numpy array so basically converting list of lists to array of arrays?

I have checked the manual.. and np.append() methods that doesn't work as for np.append() to work, it needs two arguments to append it together.

Any clues?

Answer 1

Create the 2D array up front, and fill the rows while looping:

my_array = numpy.empty((len(huge_list_of_lists), row_length))
for i, x in enumerate(huge_list_of_lists):
    my_array[i] = create_row(x)

where create_row() returns a list or 1D NumPy array of length row_length.

Depending on what create_row() does, there might be even better approaches that avoid the Python loop altogether.

Answer 2

Just pass the list of lists to numpy.array, keep in mind that numpy arrays are ndarrays, so the concept to a list of lists doesn't translate to arrays of arrays it translates to a 2d array.

>>> import numpy as np
>>> a = [[1., 2., 3.], [4., 5., 6.]]
>>> b = np.array(a)
>>> b
array([[ 1.,  2.,  3.],
       [ 4.,  5.,  6.]])
>>> b.shape
(2, 3)

Also ndarrays have nd-indexing so [1][1] becomes [1, 1] in numpy:

>>> a[1][1]
5.0
>>> b[1, 1]
5.0

Did I misunderstand your question?

You defiantly don't want to use numpy.append for something like this. Keep in mind that numpy.append has O(n) run time so if you call it n times, once for each row of your array, you end up with a O(n^2) algorithm. If you need to create the array before you know what all the content is going to be, but you know the final size, it's best to create an array using numpy.zeros(shape, dtype) and fill it in later. Similar to Sven's answer.