Regression with multi-dimensional targets

Question

I am using scikit-learn to do regression and my problem is the following. I need to do regression on several parameters (vectors). This works fine with some regression approaches such as ensemble.ExtraTreesRegressor and ensemble.RandomForestRegressor. Indeed, one can give a vector of vectors as targets to fit the model (fit(X,y) method) for the two aforementionned regression methods.

However when I try with ensemble.GradientBoostingRegressor, ensemble.AdaBoostRegressor and linear_model.SGDRegressor, the classifier fails to fit the model because it expects 1-dimensional values as targets (y argument of the fit(X,y) method). This means, with those Regression methods I can estimate only one parameter at a time. This is not suitable for my problem because it might take some time while I need to estimate about 20 parameters. On the other hand, I really would like to test those approaches.

So, my question is: Does anyone know if there is a solution to fit the model once and estimate several parameters for ensemble.GradientBoostingRegressor, ensemble.AdaBoostRegressor and linear_model.SGDRegressor?

I hope I've been clear enough ...

Answer 1

I interpret that what you have is a problem of multiple multivariate regression.

Not every regression method in scikit-learn can handle this sort of problem and you should consult the documentation of each one to find it out. In particular, neither SGDRegressor, GradientBoostingRegressor nor AdaBoostRegressor support this at the moment: fit(X, y) specifies X : array-like, shape = [n_samples, n_features] and y: array-like, shape = [n_samples].

However, you can use other methods in scikit-learn. For example, linear models:

from sklearn import linear_model
# multivariate input
X = [[0., 0.], [1., 1.], [2., 2.], [3., 3.]]
# univariate output
Y = [0., 1., 2., 3.]
# multivariate output
Z = [[0., 1.], [1., 2.], [2., 3.], [3., 4.]]

# ordinary least squares
clf = linear_model.LinearRegression()
# univariate
clf.fit(X, Y)
clf.predict ([[1, 0.]])
# multivariate
clf.fit(X, Z)
clf.predict ([[1, 0.]])

# Ridge
clf = linear_model.BayesianRidge()
# univariate
clf.fit(X, Y)
clf.predict ([[1, 0.]])
# multivariate
clf.fit(X, Z)
clf.predict ([[1, 0.]])

# Lasso
clf = linear_model.Lasso()
# univariate
clf.fit(X, Y)
clf.predict ([[1, 0.]])
# multivariate
clf.fit(X, Z)
clf.predict ([[1, 0.]])

Answer 2

As already mentioned, only some models support multivariate output. If you want to use one of the others, you can use a new class for parallelization of regressors for multivariate output: MultiOutputRegressor.

You can use it like this:

from sklearn.datasets import load_linnerud
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.multioutput import MultiOutputRegressor

linnerud = load_linnerud()

X = linnerud.data
Y = linnerud.target

# to set number of jobs to the number of cores, use n_jobs=-1
MultiOutputRegressor(GradientBoostingRegressor(), n_jobs=-1).fit(X, Y)