Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Python Numpy append array without flattening

In Python 3 I am importing several data files in a loop, and I would like to be able to store all the data in a single 2-dimensional array. I start with something like data = np.array([]) and on each iteration i want to add a new array datai = np.array([1,2,3]), how can I get my final array to look like this? [[1,2,3],[1,2,3],...,[1,2,3]]

I have tried np.append, np.concatenate, and np.stack, but none seem to work. An example code that I'm trying:

data = np.array([])
for i in range(datalen):
    datai = *func to load data as array*
    data = np.append(data, datai)

but of course this returns a flattened array. Is there any way I can get back a 2-dimensional array of length datalen with each element being the array datai?

Thanks!

like image 425
Vicente Esnaola Avatar asked Aug 03 '18 16:08

Vicente Esnaola


People also ask

Is appending to NumPy array efficient?

Appending to numpy arrays is very inefficient. This is because the interpreter needs to find and assign memory for the entire array at every single step. Depending on the application, there are much better strategies. If you know the length in advance, it is best to pre-allocate the array using a function like np.

Is Ravel same as flatten?

Ravel is faster than flatten() as it does not occupy any memory. Flatten() is comparatively slower than ravel() as it occupies memory. Ravel is a library-level function. Flatten is a method of an ndarray object.

Is NumPy concatenate faster than append?

concat function is 50 times faster than using the DataFrame. append version. With multiple append , a new DataFrame is created at each iteration, and the underlying data is copied each time.


1 Answers

The fastest way would be vstack

data = np.vstack((get_data() for i in range(datalen)))

vstack requires a tuple/iterable

data = np.vstack((data1, data2, data3))

or you can do this by appending with axis=0

data = np.empty(shape=(0, 3))
data = np.append(data, datai.reshape((-1, 3)), axis=0)  # -1 will make the rows automatic
like image 112
justengel Avatar answered Sep 18 '22 12:09

justengel