Value filter in pandas dataframe keeping NaN

Question

I am trying to filter data from a dataframe which are less than a certain value. If there is no NaN then its working fine. But when there is a nan then it is ignoring the NaN value. I want to include all the time its doesn't matter its less than or bigger than the comparing value.

import pandas as pd
import numpy as np

df = pd.DataFrame(
    {
        'index': [1, 2, 3,  4,  5,  6,   7,  8, 9],
        'value': [5, 6, 7, np.nan, 9, 3, 11, 34, 78]
    }
)

df_chunked = df[(df['index'] >= 1) & (df['index'] <= 5)]

print('df_chunked')
print(df_chunked)

df_result = df_chunked[(df_chunked['value'] < 10)]
# df_result = df_chunked[(df_chunked['value'] < 10) | (df_chunked['value'] == np.isnan(df_chunked['value']))]

print('df_result')
print(df_result)

In the above result 5,6,7,9 is showing. but i want also the nan there. I tried with

df_result = df_chunked[(df_chunked['value'] < 10) | (df_chunked['value'] == np.isnan(df_chunked['value']))]

But it is not working.

How can I do this?

Answer 1

Use not operator: ~

df_chunked[~(df_chunked['value'].ge(10))]
#df_chunked[~(df_chunked['value']>=10)] #greater or equal(the same)

   index  value
0      1    5.0
1      2    6.0
2      3    7.0
3      4    NaN
4      5    9.0

why?

Because the logical operations simply ignore NaN values and take it as False, always as you can see in the following data frame, then if you want to avoid using series.isna ( avoid unnecessary additional code) and simplify your code simply use the inverse logic with ~

print(df.assign(greater_than_5 = df['value'].gt(5),
          not_greater_than_5 = df['value'].le(5)))


   index  value  greater_than_5  not_greater_than_5
0      1    5.0           False                True
1      2    6.0            True               False
2      3    7.0            True               False
3      4    NaN           False               False
4      5    9.0            True               False
5      6    3.0           False                True
6      7   11.0            True               False
7      8   34.0            True               False
8      9   78.0            True               False

Answer 2

Try:

df_result = df_chunked[(df_chunked['value'] < 10) | (df_chunked['value'].isna())]
df_result 
   index  value
0      1    5.0
1      2    6.0
2      3    7.0
3      4    NaN
4      5    9.0