Pandas: return NaN rows

Question

I am trying to return a df that contains all of the NaN values for column == years_exp so that I can identify the corresponding id.thomas (basically I'm debugging some data that I parsed by hand). I also need to return a df with all min values. This is what I have tried so far:

rr.head(5)

    years   id.thomas   years_exp
55  2005          2     17
56  2006          2     18
57  2007          2     19
58  2008          2     20
59  2009          2     21

c = rr
c = c[c.years_exp == 'NaN']

Error:

TypeError: invalid type comparison

I'm using syntax that I copied from a youtube video on Pandas. Does anyone have an idea about the error?

Answer 1

You need isnull for checking NaN values:

print (rr[rr.years_exp.isnull()])

Docs:

Warning

One has to be mindful that in python (and numpy), the nan's don’t compare equal, but None's do. Note that Pandas/numpy uses the fact that np.nan != np.nan, and treats None like np.nan.

In [11]: None == None
Out[11]: True

In [12]: np.nan == np.nan
Out[12]: False

So as compared to above, a scalar equality comparison versus a None/np.nan doesn’t provide useful information.

In [13]: df2['one'] == np.nan
Out[13]: 
a    False
b    False
c    False
d    False
e    False
f    False
g    False
h    False
Name: one, dtype: bool

Answer 2

You can try with

c = c.loc[c.years_exp == 'NaN']

or

c = c.loc[c.years_exp == None]

or

c = c.loc[c.years_exp.isnull()]