I have the following code snippet from a program called Flights.py
...
#Load the Dataset
df = dataset
df.isnull().any()
df = df.fillna(lambda x: x.median())
# Define X and Y
X = df.iloc[:, 2:124].values
y = df.iloc[:, 136].values
X_tolist = X.tolist()
# Splitting the dataset into the Training set and Test set
from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 0)
# Feature Scaling
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)
The second to last line is throwing the following error:
Traceback (most recent call last):
File "<ipython-input-14-d4add2ccf5ab>", line 3, in <module>
X_train = sc.fit_transform(X_train)
File "/Users/<username>/anaconda/lib/python3.6/site-packages/sklearn/base.py", line 494, in fit_transform
return self.fit(X, **fit_params).transform(X)
File "/Users/<username>/anaconda/lib/python3.6/site-packages/sklearn/preprocessing/data.py", line 560, in fit
return self.partial_fit(X, y)
File "/Users/<username>/anaconda/lib/python3.6/site-packages/sklearn/preprocessing/data.py", line 583, in partial_fit
estimator=self, dtype=FLOAT_DTYPES)
File "/Users/<username>/anaconda/lib/python3.6/site-packages/sklearn/utils/validation.py", line 382, in check_array
array = np.array(array, dtype=dtype, order=order, copy=copy)
TypeError: float() argument must be a string or a number, not 'function'
My dataframe df
is of size (22587, 138)
I was taking a look at the following question for inspiration:
TypeError: float() argument must be a string or a number, not 'method' in Geocoder
I tried the following adjustment:
# Feature Scaling
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train.as_matrix)
X_test = sc.transform(X_test.as_matrix)
Which resulted in the following error:
AttributeError: 'numpy.ndarray' object has no attribute 'as_matrix'
I'm currently at a loss for how to scan thru the dataframe and find/convert the offending entries.
As this answer explains, fillna
isn't designed to work with a callback. If you pass one, it will be taken as the literal fill value, meaning your NaN
s will be replaced with lambdas:
df
col1 col2 col3 col4
row1 65.0 24 47.0 NaN
row2 33.0 48 NaN 89.0
row3 NaN 34 67.0 NaN
row4 24.0 12 52.0 17.0
df4.fillna(lambda x: x.median())
col1 col2 \
row1 65 24
row2 33 48
row3 <function <lambda> at 0x10bc47730> 34
row4 24 12
col3 col4
row1 47 <function <lambda> at 0x10bc47730>
row2 <function <lambda> at 0x10bc47730> 89
row3 67 <function <lambda> at 0x10bc47730>
row4 52 17
If you are trying to fill by median, the solution would be to create a dataframe of medians based on the column, and pass that to fillna
.
df
col1 col2 col3 col4
row1 65.0 24 47.0 NaN
row2 33.0 48 NaN 89.0
row3 NaN 34 67.0 NaN
row4 24.0 12 52.0 17.0
df.fillna(df.median())
df
col1 col2 col3 col4
row1 65.0 24 47.0 53.0
row2 33.0 48 52.0 89.0
row3 33.0 34 67.0 53.0
row4 24.0 12 52.0 17.0
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With