Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Numpy inconsistent results with Pandas and missing values

Why does numpy return different results with missing values when using a Pandas series compared to accessing the series' values as in the following:

import pandas as pd
import numpy as np

data = pd.DataFrame(dict(a=[1, 2, 3, np.nan, np.nan, 6]))
np.sum(data['a'])

#12.0

np.sum(data['a'].values)
#nan
like image 919
willk Avatar asked Mar 12 '19 20:03

willk


1 Answers

Calling np.sum on a pandas Series delegates to Series.sum, which ignores NaNs when computing the sum (BY DEFAULT).

data['a'].sum()
# 12.0

np.sum(data['a'])
# 12.0

You can see this from the source code of np.sum:

np.sum??

def sum(a, axis=None, dtype=None, out=None, keepdims=np._NoValue, initial=np._NoValue):
    ...
    return _wrapreduction(a, np.add, 'sum', axis, dtype, out, keepdims=keepdims,

Taking a look at the source code for _wrapreduction, we see:

np.core.fromnumeric._wrapreduction??

def _wrapreduction(obj, ufunc, method, axis, dtype, out, **kwargs):
    ...
    if type(obj) is not mu.ndarray:
        try:
            reduction = getattr(obj, method)   # get reference to Series.add 

reduction is then finally called at the end of the function:

            return reduction(axis=axis, out=out, **passkwargs)           
like image 187
cs95 Avatar answered Nov 01 '22 07:11

cs95