Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to extend an array in-place in Numpy?

Currently, I have some code like this

import numpy as np ret = np.array([]) for i in range(100000):   tmp =  get_input(i)   ret = np.append(ret, np.zeros(len(tmp)))   ret = np.append(ret, np.ones(fixed_length)) 

I think this code is not efficient as np.append needs to return a copy of the array instead of modify the ret in-place

I was wondering whether I can use the extend for a numpy array like this:

import numpy as np from somewhere import np_extend ret = np.array([]) for i in range(100000):   tmp =  get_input(i)   np_extend(ret, np.zeros(len(tmp)))   np_extend(ret, np.ones(fixed_length)) 

So that the extend would be much more efficient. Does anyone have ideas about this? Thanks!

like image 560
Hanfei Sun Avatar asked Nov 04 '12 02:11

Hanfei Sun


People also ask

How do you make an array longer in Python?

resize() can create an array of larger size than the original array. To convert the original array into a bigger array, resize() will add more elements (than available in the original array) by copying the existing elements (repeated as many times as required) to fill the desired larger size.

How do I change the length of a NumPy array?

there is no converting the dimensions of a numpy array in python. A numpy array is simply a section of your RAM. You can't append to it in the sense of literally adding bytes to the end of the array, but you can create another array and copy over all the data (which is what np. append(), or np.

Is NumPy append in place?

Numpy with PythonThe append operation is not inplace, a new array is allocated. Also the dimensions of the input arrays must match otherwise ValueError will be generated.

How do I increase the dimensions of an array in NumPy?

You can add new dimensions to a NumPy array ndarray (= unsqueeze a NumPy array) with np. newaxis , np. expand_dims() and np. reshape() (or reshape() method of ndarray ).


2 Answers

Imagine a numpy array as occupying one contiguous block of memory. Now imagine other objects, say other numpy arrays, which are occupying the memory just to the left and right of our numpy array. There would be no room to append to or extend our numpy array. The underlying data in a numpy array always occupies a contiguous block of memory.

So any request to append to or extend our numpy array can only be satisfied by allocating a whole new larger block of memory, copying the old data into the new block and then appending or extending.

So:

  1. It will not occur in-place.
  2. It will not be efficient.
like image 121
unutbu Avatar answered Sep 30 '22 16:09

unutbu


You can use the .resize() method of ndarrays. It requires that the memory is not referred to by other arrays/variables.

import numpy as np ret = np.array([]) for i in range(100):     tmp = np.random.rand(np.random.randint(1, 100))     ret.resize(len(ret) + len(tmp)) # <- ret is not referred to by anything else,                                     #    so this works     ret[-len(tmp):] = tmp 

The efficiency can be improved by using the usual array memory overrallocation schemes.

like image 41
pv. Avatar answered Sep 30 '22 17:09

pv.