How to disregard the NaN data point in numpy array and generate the normalized data in Python?

Question

Say I have a numpy array that has some float('nan'), I don't want to impute those data now and I want to first normalize those and keep the NaN data at the original space, is there any way I can do that?

Previously I used normalize function in sklearn.Preprocessing, but that function seems can't take any NaN contained array as input.

Answer 1

You can mask your array using the numpy.ma.array function and subsequently apply any numpy operation:

import numpy as np

a = np.random.rand(10)            # Generate random data.
a = np.where(a > 0.8, np.nan, a)  # Set all data larger than 0.8 to NaN

a = np.ma.array(a, mask=np.isnan(a)) # Use a mask to mark the NaNs

a_norm  = a / np.sum(a) # The sum function ignores the masked values.
a_norm2 = a / np.std(a) # The std function ignores the masked values.

You can still access your raw data:

print a.data

Answer 2

You can use numpy.nansum to compute the norm and ignore nan:

In [54]: x
Out[54]: array([  1.,   2.,  nan,   3.])

Here's the norm with nan ignored:

In [55]: np.sqrt(np.nansum(np.square(x)))
Out[55]: 3.7416573867739413

y is the normalized array:

In [56]: y = x / np.sqrt(np.nansum(np.square(x)))

In [57]: y
Out[57]: array([ 0.26726124,  0.53452248,         nan,  0.80178373])

In [58]: np.linalg.norm(y[~np.isnan(y)])
Out[58]: 1.0