Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Python pandas apply function if a column value is not NULL

I have a dataframe (in Python 2.7, pandas 0.15.0):

df=        A    B               C 0    NaN   11             NaN 1    two  NaN  ['foo', 'bar'] 2  three   33             NaN 

I want to apply a simple function for rows that does not contain NULL values in a specific column. My function is as simple as possible:

def my_func(row):     print row 

And my apply code is the following:

df[['A','B']].apply(lambda x: my_func(x) if(pd.notnull(x[0])) else x, axis = 1) 

It works perfectly. If I want to check column 'B' for NULL values the pd.notnull() works perfectly as well. But if I select column 'C' that contains list objects:

df[['A','C']].apply(lambda x: my_func(x) if(pd.notnull(x[1])) else x, axis = 1) 

then I get the following error message: ValueError: ('The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()', u'occurred at index 1')

Does anybody know why pd.notnull() works only for integer and string columns but not for 'list columns'?

And is there a nicer way to check for NULL values in column 'C' instead of this:

df[['A','C']].apply(lambda x: my_func(x) if(str(x[1]) != 'nan') else x, axis = 1) 

Thank you!

like image 993
ragesz Avatar asked Oct 28 '14 17:10

ragesz


People also ask

How do you check if a column is not null in pandas?

Python Pandas – Check for Null values using notnull() Now, on displaying the DataFrame, the CSV data will be displayed in the form of True and False i.e. boolean values because notnull() returns boolean. For Null values, False will get displayed. For Not-Null values, True will get displayed.

IS NOT null function pandas?

notnull is a pandas function that will examine one or multiple values to validate that they are not null. In Python, null values are reflected as NaN (not a number) or None to signify no data present. . notnull will return False if either NaN or None is detected. If these values are not present, it will return True.

How do you filter a pandas DataFrame based on null values of a column?

You can filter out rows with NAN value from pandas DataFrame column string, float, datetime e.t.c by using DataFrame. dropna() and DataFrame. notnull() methods. Python doesn't support Null hence any missing data is represented as None or NaN.


2 Answers

The problem is that pd.notnull(['foo', 'bar']) operates elementwise and returns array([ True, True], dtype=bool). Your if condition trys to convert that to a boolean, and that's when you get the exception.

To fix it, you could simply wrap the isnull statement with np.all:

df[['A','C']].apply(lambda x: my_func(x) if(np.all(pd.notnull(x[1]))) else x, axis = 1) 

Now you'll see that np.all(pd.notnull(['foo', 'bar'])) is indeed True.

like image 89
Korem Avatar answered Sep 23 '22 09:09

Korem


I had a column contained lists and NaNs. So, the next one worked for me.

df.C.map(lambda x: my_func(x) if type(x) == list else x) 
like image 24
coffman21 Avatar answered Sep 23 '22 09:09

coffman21