Cross Validation in linear regression

Question

I am trying to perform cross validation in Linear Regression, for which I am using python sklearn libraries. I have a question regarding the appropriate way of performing cross validation for a given dataset.

The two APIs that are confusing me a bit are cross_val_score() and any regularized cross validation algorithm, like LassoCV().

As I understand, cross_val_score is used to get the score based on cross validation. And, it can be clubbed with Lasso() to achieve regularized cross validation score (Example: here).

In contrast, LassoCV(), as it's documentation suggests, performs Lassofor a given range of tuning parameter (alpha or lambda).

Now, my questions are:

• Which one is a better approach (cross_val_score with Lasso or just LassoCV).
• What is the correct way of performing cross validation for Linear Regression (or other algorithms, say Logistic, NN etc.)

Thanks.

Answer 1

To confuse you a bit more - consider using GridSearchCV, which will do cross validation and tune up hyperparameters.

Demo:

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import Lasso, Ridge, SGDRegressor
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import Pipeline, FeatureUnion

X_train, X_test, y_train, y_test = \
        train_test_split(X, y, test_size = 0.33)

pipe = Pipeline([
    ('scale', StandardScaler()),
    ('regr', Lasso())
])

param_grid = [
    {
        'regr': [Lasso(), Ridge()],
        'regr__alpha': np.logspace(-4, 1, 6),
    },
    {
        'regr': [SGDRegressor()],
        'regr__alpha': np.logspace(-5, 0, 6),
        'regr__max_iter': [500, 1000],
    },
]

grid = GridSearchCV(pipe, param_grid=param_grid, cv=3, n_jobs=-1, verbose=2)
grid.fit(X_train, y_train)

predicted = grid.predict(X_test, y_test)

print('Score:\t{}'.format(grid.score(X_test, y_test)))