Python\Numpy: Comparing arrays with NAN [duplicate]

Question

Why are the following two lists not equal?

a = [1.0, np.NAN] 
b = np.append(np.array(1.0), [np.NAN]).tolist()

I am using the following to check for identicalness.

((a == b) | (np.isnan(a) & np.isnan(b))).all(), np.in1d(a,b)

Using np.in1d(a, b) it seems the np.NAN values are not equal but I am not sure why this is. Can anyone shed some light on this issue?

Answer 1

NaN values never compare equal. That is, the test NaN==NaN is always False by definition of NaN.

So [1.0, NaN] == [1.0, NaN] is also False. Indeed, once a NaN occurs in any list, it cannot compare equal to any other list, even itself.

If you want to test a variable to see if it's NaN in numpy, you use the numpy.isnan() function. I don't see any obvious way of obtaining the comparison semantics that you seem to want other than by “manually” iterating over the list with a loop.

Consider the following:

import math
import numpy as np

def nan_eq(a, b):
    for i,j in zip(a,b):
        if i!=j and not (math.isnan(i) and math.isnan(j)):
            return False
    return True

a=[1.0, float('nan')]
b=[1.0, float('nan')]

print( float('nan')==float('nan') )
print( a==a )
print( a==b )
print( nan_eq(a,a) )

It will print:

False
True
False
True

The test a==a succeeds because, presumably, Python's idea that references to the same object are equal trumps what would be the result of the element-wise comparison that a==b requires.

Answer 2

Since a and b are lists, a == b isn't returning an array, and so your numpy-like logic won't work:

>>> a == b
False

The command you've quoted only works if they're arrays:

>>> a,b = np.asarray(a), np.asarray(b)
>>> a == b
array([ True, False], dtype=bool)
>>> (a == b) | (np.isnan(a) & np.isnan(b))
array([ True,  True], dtype=bool)
>>> ((a == b) | (np.isnan(a) & np.isnan(b))).all()
True

which should work to compare two arrays (either they're both equal or they're both NaN).