In Python 3 I am importing several data files in a loop, and I would like to be able to store all the data in a single 2-dimensional array. I start with something like data = np.array([])
and on each iteration i want to add a new array datai = np.array([1,2,3])
, how can I get my final array to look like this? [[1,2,3],[1,2,3],...,[1,2,3]]
I have tried np.append, np.concatenate, and np.stack, but none seem to work. An example code that I'm trying:
data = np.array([])
for i in range(datalen):
datai = *func to load data as array*
data = np.append(data, datai)
but of course this returns a flattened array. Is there any way I can get back a 2-dimensional array of length datalen
with each element being the array datai
?
Thanks!
Appending to numpy arrays is very inefficient. This is because the interpreter needs to find and assign memory for the entire array at every single step. Depending on the application, there are much better strategies. If you know the length in advance, it is best to pre-allocate the array using a function like np.
Ravel is faster than flatten() as it does not occupy any memory. Flatten() is comparatively slower than ravel() as it occupies memory. Ravel is a library-level function. Flatten is a method of an ndarray object.
concat function is 50 times faster than using the DataFrame. append version. With multiple append , a new DataFrame is created at each iteration, and the underlying data is copied each time.
The fastest way would be vstack
data = np.vstack((get_data() for i in range(datalen)))
vstack requires a tuple/iterable
data = np.vstack((data1, data2, data3))
or you can do this by appending with axis=0
data = np.empty(shape=(0, 3))
data = np.append(data, datai.reshape((-1, 3)), axis=0) # -1 will make the rows automatic
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With