Python error cannot do a non empty take from an empty axes

Question

I have a pandas dataframe with more than 400 thousands rows and now I want to calculate the interquartile range for each row but my code produced the following errors:

cannot do a non empty take from an empty axes

My code:

def calIQR(x):
    x=x.dropna()
    return (np.percentile(x,75),np.percentile(x,25))

df["count"]=df.iloc[:,2:64].apply(calIQR,axis=1)

I am running Python 2.7.13

I searched online but still had no idea why this error occurred.

The 2 to 64 columns of dataset basically look like that:

In each row, there are some NaN values, but I am sure that there is no row will all NaN.

Answer 1

I think here is problem row has all NaNs values in 2 to 63 columns and x = x.dropna return empty Series.

So need add dropna after iloc:

np.random.seed(100)
df = pd.DataFrame(np.random.random((5,5)))
df.loc[3, [3,4]] = np.nan
df.loc[2] = np.nan
print (df)
         0         1         2         3         4
0  0.543405  0.278369  0.424518  0.844776  0.004719
1  0.121569  0.670749  0.825853  0.136707  0.575093
2       NaN       NaN       NaN       NaN       NaN
3  0.978624  0.811683  0.171941       NaN       NaN
4  0.431704  0.940030  0.817649  0.336112  0.175410

---

def calIQR(x):
    x = x.dropna()
    return (np.percentile(x,75),np.percentile(x,25))

df["count"]=df.iloc[:,2:4].dropna(how='all').apply(calIQR,axis=1)
print (df)
          0         1         2         3         4  \
0  0.543405  0.278369  0.424518  0.844776  0.004719   
1  0.121569  0.670749  0.825853  0.136707  0.575093   
2       NaN       NaN       NaN       NaN       NaN   
3  0.978624  0.811683  0.171941       NaN       NaN   
4  0.431704  0.940030  0.817649  0.336112  0.175410   

                              count  
0  (0.739711496927, 0.529582226142)  
1    (0.65356621375, 0.30899313104)  
2                               NaN  
3  (0.171941012733, 0.171941012733)  
4  (0.697265021613, 0.456496307285)  

Or use Series.quantile:

 def calIQR(x):
    return (x.quantile(.75),x.quantile(.25))

#with real data change 2;4 to 2:64
df["count"]=df.iloc[:,2:4].apply(calIQR,axis=1)
print (df)
          0         1         2         3         4  \
0  0.543405  0.278369  0.424518  0.844776  0.004719   
1  0.121569  0.670749  0.825853  0.136707  0.575093   
2       NaN       NaN       NaN       NaN       NaN   
3  0.978624  0.811683  0.171941       NaN       NaN   
4  0.431704  0.940030  0.817649  0.336112  0.175410   

                                       count  
0   (0.7397114969272109, 0.5295822261418257)  
1    (0.653566213750024, 0.3089931310399766)  
2                                 (nan, nan)  
3   (0.1719410127325942, 0.1719410127325942)  
4  (0.6972650216127702, 0.45649630728485585)