Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Check for None in pandas dataframe

I would like to find where None is found in the dataframe.

pd.DataFrame([None,np.nan]).isnull()
OUT: 
      0
0  True
1  True

isnull() finds both numpy Nan and None values.

I only want the None values and not numpy Nan. Is there an easier way to do that without looping through the dataframe?

Edit: After reading the comments, I realized that in my dataframe in my work also include strings, so the None were not coerced to numpy Nan. So the answer given by Pisdom works.

like image 484
ConanG Avatar asked Jul 24 '17 01:07

ConanG


People also ask

How to check if any value is Nan in a pandas Dataframe?

How to Check If Any Value is NaN in a Pandas DataFrame. The official documentation for pandas defines what most developers would know as null values as missing or missing data in pandas. Within pandas, a missing value is denoted by NaN. In most cases, the terms missing and null are interchangeable, but to abide by the standards of pandas, ...

How to detect missing values in pandas Dataframe?

Pandas isnull () function detect missing values in the given object. It return a boolean same-sized object indicating if the values are NA. Missing values gets mapped to True and non-missing value gets mapped to False. Return Type: Dataframe of Boolean values which are True for NaN values otherwise False.

How to check nulls in pandas?

Pandas is proving two methods to check NULLs - isnull() and notnull() These two returns TRUE and FALSE respectively if the value is . So let's check what it will return for our data. isnull() test. notnull() test. Check 0th row, LoanAmount Column - In isnull() test it is TRUE and in notnull() test it is FALSE.

What are returns in pandas Dataframe?

Returns : sum of Series or DataFrame (if level specified). Let’s create a pandas dataframe. Example 1 : Count total NaN at each column in DataFrame.


1 Answers

If you want to get True/False for each line, you can use the following code. Here is an example as a result for the following DataFrame:

df = pd.DataFrame([[None, 3], ["", np.nan]])

df
#      0      1
#0  None    3.0
#1          NaN

How to check None

Available: .isnull()

>>> df[0].isnull()
0     True
1    False
Name: 0, dtype: bool

Available: .apply == or is None

>>> df[0].apply(lambda x: x == None)
0     True
1    False
Name: 0, dtype: bool

>>> df[0].apply(lambda x: x is None)
0     True
1    False
Name: 0, dtype: bool

Available: .values == None

>>> df[0].values == None
array([ True, False])

Unavailable: is or ==

>>> df[0] is None
False

>>> df[0] == None
0    False
1    False
Name: 0, dtype: bool

Unavailable: .values is None

>>> df[0].values is None
False

How to check np.nan

Available: .isnull()

>>> df[1].isnull()
0    False
1     True
Name: 1, dtype: bool

Available: np.isnan

>>> np.isnan(df[1])
0    False
1     True
Name: 1, dtype: bool

>>> np.isnan(df[1].values)
array([False,  True])

>>> df[1].apply(lambda x: np.isnan(x))
0    False
1     True
Name: 1, dtype: bool

Unavailable: is or == np.nan

>>> df[1] is np.nan
False

>>> df[1] == np.nan
0    False
1    False
Name: 1, dtype: bool

>>> df[1].values is np.nan
False

>>> df[1].values == np.nan
array([False, False])

>>> df[1].apply(lambda x: x is np.nan)
0    False
1    False
Name: 1, dtype: bool

>>> df[1].apply(lambda x: x == np.nan)
0    False
1    False
Name: 1, dtype: bool
like image 193
Keiku Avatar answered Oct 10 '22 02:10

Keiku