Python: memory error while changing data type from integer to float

Question

I have a array of size 13000*300000 filled with integer from 0 to 255. I would like to change their data type from integer to float as if data is a numpy array:

 data.astype('float')

While changing its data type from integer to float, it shows memory error. I have 80 GB of RAM. It still shows memory error. Could you please let me know what can be the reason for it?

Answer 1

The problem here is that data is huge (about 30GB of sequential data, see How much memory in numpy array?), hence it causes the error while trying to fit it into the memory. Instead of doing the operation on whole, slice it and then do the operation and then merge, like:

n = 300000
d1 = data[:, :n/2].astype('float')
d2 = data[:, n/2:].astype('float')

data = np.hstack(d1, d2)

Generally, since your data size is so unwieldy, consider consuming it in parts to avoid being bitten by these sorts of problems all the time (see Techniques for working with large Numpy arrays? for this and other techniques).