Standardization/Normalization test data in Python

Question

I'm working in a sklearn homework and I don't understand why one should standardize and normalize the test data with the training mean and sd. How can I implement this in Python? Here is my implementation for the train data:

digits = sklearn.datasets.load_digits()
X= digits.data
Y= digits.target
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.3,train_size=0.7)
std_scale = preprocessing.StandardScaler().fit(X_train)
X_train_std = std_scale.transform(X_train)
#X_test_std=??

For the train i think it's correct, but for the test?

Answer 1

Why?

Because your classifier/regressor will be trained on those standardizes values. You don't want to use your trained-classifier to predict data which has other statistics.

How:

std_scale = preprocessing.StandardScaler().fit(X_train)
X_train_std = std_scale.transform(X_train)
X_test_std  = std_scale.transform(X_test)

Fitting once, transforming whatever you need to transform. That's the advantage of the class-based StandardScaler (which you already had chosen) compared to scale which does not hold the needed information needed for applying transformations (based on these statistics obtained during fit) at a later time.