import pandas as pd
df = pd.read_csv('file.csv')
df.dropna(inplace=True)
filter1 = df['col1'] == 'some_value'
filter2 = df['col2'] == 'some_other_value'
df.where(filter1 & filter2, inplace=True)
df.head()
localCountry localState remoteCountry remoteState ... col1 col2 col3 num_samples
1250 NaN NaN NaN NaN ... NaN NaN NaN NaN
1251 NaN NaN NaN NaN ... NaN NaN NaN NaN
I thought dropna()
would remove all rows that contained at least one NaN
. Why is there NaN in the result here? A misunderstanding on my part for sure, but I haven't been able to logic out why this is the case after using dropna
prior to the where
method.
Edit for others:
The where()
method replaces the values where the condition passed is false. If you don't supply a default, it will replace them with NaN. It is not just a query to return rows where both conditions are met.
Dataframe.where
Where cond is True, keep the original value. Where False, replace with corresponding value from other. If cond is callable, it is computed on the Series/DataFrame and should return boolean Series/DataFrame or array. The callable must not change input Series/DataFrame (though pandas doesn’t check it).
I think the problem is you are missing the default value of DataFrame.where
for setting rows not matched by any of the conditions (other
):
df = pd.DataFrame({'col1':['some_value','some_value','aaa','dd'],
'col2':['some_other_value','dd','some_other_value', 'bb'],
'col3':list('abcd')})
print (df)
col1 col2 col3
0 some_value some_other_value 0
1 some_value dd 1
2 aaa some_other_value 2
3 dd bb 3
filter1 = df['col1'] == 'some_value'
filter2 = df['col2'] == 'some_other_value'
df.where(filter1 & filter2, other=np.nan, inplace=True)
print (df)
0 some_value some_other_value a
1 NaN NaN NaN
2 NaN NaN NaN
3 NaN NaN NaN
If you change the replacement value:
df.where(filter1 & filter2, other='val', inplace=True)
col1 col2 col3
0 some_value some_other_value a
1 val val val
2 val val val
3 val val val
If you want to filter the rows, then use boolean indexing
:
df1 = df[filter1 & filter2]
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With