target scaling using GridSearchCV

Question

For hyperparameter tuning, I use the function GridSearchCV from the Python package sklearn. Some of the models that I test require feature scaling (e.g. Support Vector Regression - SVR). Recently, in the Udemy course Machine Learning A-Z™: Hands-On Python & R In Data Science, the instructors mentioned that for SVR, the target should also be scaled (if it is not binary). Bearing this in mind, I wonder whether the target is also scaled in each iteration of the cross-validation procedure performed by GridSearchCV or if only the features are scaled. Please see the code below, which illustrates the normal procedure that I use for hyperparameter tuning for estimators that require scaling for the training sets:

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import GridSearchCV
from sklearn.svm import SVR
    
def SVRegressor(**kwargs):
   '''contruct a pipeline to perform SVR regression'''
   return make_pipeline(StandardScaler(), SVR(**kwargs))

params = {'svr__kernel': ["poly", "rbf"]}
grid_search = GridSearchCV(SVRegressor(), params)
grid_search.fit(X, y)

I know that I could simply scale X and y a priori and drop the StandardScaler from the pipeline. However, I want to implement this approach in a code pipeline where multiple models are tested, in which, some require scaling and others do not. That is why I want to know how GridSearchCV handles scaling under the hood.

Answer 1

No it doesn't scale the target, if you look at make_pipeline, it simply passes the X and y argument into your transformer, and StandardScaler() does nothing to your y:

def _fit_transform_one(transformer,
                       X,
                       y,
                       weight,
                       message_clsname='',
                       message=None,
                       **fit_params):
    """
    Fits ``transformer`` to ``X`` and ``y``. The transformed result is returned
    with the fitted transformer. If ``weight`` is not ``None``, the result will
    be multiplied by ``weight``.
    """
    with _print_elapsed_time(message_clsname, message):
        if hasattr(transformer, 'fit_transform'):
            res = transformer.fit_transform(X, y, **fit_params)
        else:
            res = transformer.fit(X, y, **fit_params).transform(X)

    if weight is None:
        return res, transformer
    return res * weight, transformer

You can try this on StandardScaler() and you can see it does not do anything with y:

np.random.seed(111)
X = np.random.normal(5,2,(100,3))
y = np.random.normal(5,2,100)

res = StandardScaler().fit_transform(X=X,y=y)
res.shape
(100, 3)

res.mean(axis=0)
array([1.01030295e-15, 4.39648318e-16, 8.91509089e-16])

res.std(axis=0)
array([1., 1., 1.])

You can also check the result of your gridsearchcv:

SVRegressor = make_pipeline(StandardScaler(), SVR())
params = {'svr__kernel': ["poly", "rbf"]}
grid_search = GridSearchCV(SVRegressor, params,
scoring='neg_mean_absolute_error')

On unscaled y, you will see that on the unscaled data, your negative mean absolute error is around the same scale as your standard deviation (I used 2 in my example):

grid_search.fit(X, y)

grid_search.cv_results_['mean_test_score']
array([-2.01029707, -1.88779205])

On scaled y, our standard deviation would be 1, and you can see the error is around -1,:

y_scaled = StandardScaler().fit_transform(y.reshape(-1,1)).ravel()
grid_search.fit(X, y_scaled)

grid_search.cv_results_['mean_test_score']
array([-1.00585999, -0.88330208])