Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Impute entire DataFrame (all columns) using Scikit-learn (sklearn) without iterating over columns

I want to impute all of the columns on a pandas DataFrame...the only way I can think of doing this is column by column as shown below...

Is there an operation where I can impute the entire DataFrame without iterating through the columns?

#!/usr/bin/python
from sklearn.preprocessing import Imputer
import numpy as np
import pandas as pd

#Imputer
fill_NaN = Imputer(missing_values=np.nan, strategy='mean', axis=1)

#Model 1
DF = pd.DataFrame([[0,1,np.nan],[2,np.nan,3],[np.nan,2,5]])
DF.columns = "c1.c2.c3".split(".")
DF.index = "i1.i2.i3".split(".")

#Impute Series
imputed_DF = DF
for col in DF.columns:
    imputed_column = fill_NaN.fit_transform(DF[col]).T
    #Fill in Series on DataFrame
    imputed_DF[col] = imputed_column

#DF
#c1  c2  c3
#i1   0   1 NaN
#i2   2 NaN   3
#i3 NaN   2   5

#imputed_DF
#c1   c2  c3
#i1   0  1.0   4
#i2   2  1.5   3
#i3   1  2.0   5
like image 682
O.rka Avatar asked Nov 11 '15 22:11

O.rka


2 Answers

If you want the mean or median you could do something like:

fill_NaN = Imputer(missing_values=np.nan, strategy='mean', axis=1)
imputed_DF = pd.DataFrame(fill_NaN.fit_transform(DF))
imputed_DF.columns = DF.columns
imputed_DF.index = DF.index

If you want to fill them with 0s or something you could always just do:

DF[DF.isnull()] = 0
like image 88
O.rka Avatar answered Nov 09 '22 01:11

O.rka


Unless you specifically need to use the sklearn Imputer for some reason, it seems to me that a simpler option would be to just do:

df = df.fillna(df.mean())
like image 7
Biggus Avatar answered Nov 09 '22 01:11

Biggus