Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

dropping rows from dataframe based on a "not in" condition [duplicate]

Tags:

python

pandas

People also ask

How do you drop rows in a Dataframe based on multiple conditions?

We can drop single or multiple columns from the dataframe just by passing the name of columns and by setting up the axis =1.

How do you drop rows with certain conditions?

Use pandas. DataFrame. drop() method to delete/remove rows with condition(s).

How can I get the rows of dataframe1 which are not in dataframe2?

First, we need to modify the original DataFrame to add the row with data [3, 10]. Perform a left-join, eliminating duplicates in df2 so that each row of df1 joins with exactly 1 row of df2 . Use the parameter indicator to return an extra column indicating which table the row was from.


You can use pandas.Dataframe.isin.

pandas.Dateframe.isin will return boolean values depending on whether each element is inside the list a or not. You then invert this with the ~ to convert True to False and vice versa.

import pandas as pd

a = ['2015-01-01' , '2015-02-01']

df = pd.DataFrame(data={'date':['2015-01-01' , '2015-02-01', '2015-03-01' , '2015-04-01', '2015-05-01' , '2015-06-01']})

print(df)
#         date
#0  2015-01-01
#1  2015-02-01
#2  2015-03-01
#3  2015-04-01
#4  2015-05-01
#5  2015-06-01

df = df[~df['date'].isin(a)]

print(df)
#         date
#2  2015-03-01
#3  2015-04-01
#4  2015-05-01
#5  2015-06-01

You can use Series.isin:

df = df[~df.datecolumn.isin(a)]

While the error message suggests that all() or any() can be used, they are useful only when you want to reduce the result into a single Boolean value. That is however not what you are trying to do now, which is to test the membership of every values in the Series against the external list, and keep the results intact (i.e., a Boolean Series which will then be used to slice the original DataFrame).

You can read more about this in the Gotchas.