Can't find nan entries using numpy in array of strings

Question

Can't find nan entries using numpy in array of strings my code is:

for x in X_cat:
    if x == np.nan:
        print('Found')

I know for a fact there are 2 nan entries inn the list but the code runs without printing anything. same if I replace np.nan with 'nan' My final objective is to replace the nan with the most common string.

Answer 1

That's because comparing anything with NaN, including NaN, is False. So even when x is np.nan, the print will not run. (In fact that used to be an acceptable way of checking if something was NaN as no other IEEE754 floating point value has that property.)

Use np.isnan(x) to check if x is NaN.

Answer 2

In an array of strings, you can only perform string comparisons. You have to initialize a nan in a string format.

nan_str = str_np.array([np.nan]).astype(str)[0]

And by initializing an array like you describe it :

x = np.array(['hello', np.nan, 'world', np.nan], dtype=object)

You can then replace these nan by the most common string that I assume to be mostcommonstring :

x[np.where(x.astype(str)==str_nan)]='mostcommonstring'

Answer 3

You need to check x for NaN with np.isnan:

for x in X_cat:
    if np.isnan(x):
        print('Found')

np.nan == np.nan returns False, so direct comparison is meaningless here. Find more about isnan in numpy docs

Answer 4

Not enough reputation to comment on Thibaut's answer, but to simplify it: The nan-string can be np.str_(np.nan) or even str(np.nan).

x = np.array(['hello', np.nan, 'world', np.nan], dtype=object)

x[np.where(x.astype(str)==str(np.nan))] = 'mostcommonstring'