Plot partial dependency with model built from pandas data frame

Question

I have a model that is trained from a pandas dataframe. It can predict dataframe input without problem:

from xgboost import XGBClassifier
clf = XGBClassifier()
clf = clf.fit(X_train, y_train) # X_train is a pandas dataFrame with 5 columns: a,b,c,d,e.
clf.predict_proba(X_train) 

However, when I use the exact data and model to plot the partial dependency graph, I have the following error:

ValueError: feature_names mismatch: ['a', 'b', 'c', 'd', 'e'] ['f0', 'f1', 'f2', 'f3', 'f4']
expected b, a, d, c, e in input data
training data did not have the following fields: f2, f3, f1, f0, f4

The code I used was:

plot_partial_dependence(estimator=clf, X=X_train, features=[0,1])

I understand that I can convert X_train to numpy.ndarray before training the model, and it solves the problem. However, as the actual classifier is very large and it took a long time to train already, I would like to re-use the classifier that was trained with pandas dataframe.

Is there a way to do that? Thank you very much!

Edit the OP to include some sample data:

X_train.head(10):

    a        b        c    d           e
0  34   226830  5249738  409  1186.78850
1  36    38940  8210911   76  2326.72880
2  36    38940  8210911   76  2326.72880
3  34   761188  5074516  698   370.27365
4  36  1097060  9072727  296   576.91693
5  36  1097060  9072727  296   576.91693
6  25    62240   881740  102   194.59651
7  25    62240   881740  102   194.59651
8  25    62240   881740  102   194.59651
9  28    65484  1391620  105   259.25095

y_train.head(10):

0    1
1    1
2    1
3    1
4    1
5    1
6    1
7    1
8    1
9    1

Answer 1

Congrats! You found a deficiency between sklearn and xgboost.

Using the traceback to guide me, I stuck a print(data.feature_names) as the first line in Booster._validate_features. When I run your method (with dummy data I created), I get output like this:

['a', 'b', 'c', 'd', 'e']
.
.
.
['a', 'b', 'c', 'd', 'e']
['f0', 'f1', 'f2', 'f3', 'f4']

The first several lines where the feature names are correct are from fitting the model. When fitting, apparently, it is possible to set the feature names. The last line is from calling the plot_partial_dependence. Seemingly, there is no way for sklearn to propagate the column names to xgboost using this method and so the latter defaults to 'f0', 'f1', etc.

WARNING: I am uncertain if disabling feature validation in the manner described below has adverse affects (namely, that feature names are confused). It is hard to tell when using dummy data as I have. Take the resulting partial dependence plots with a grain of salt. You may want to check the results from XGBClassifier against those from sklearn's GradientBoostingClassifier as a precaution. Or, rename the columns to ['f0', 'f1', 'f2', 'f3', 'f4'] prior to training.

On the plus side, you can get around this without having to change your column names. Ideally, the plot_partial_dependence function would allow us to specify a list of keyword arguments to pass to the response_method (i.e. clf.predict_proba) because we would ideally pass validate_features=False. In lieu of this interface, I propose the following hack:

# store keyword argument default values
tmpdefaults = XGBClassifier.predict_proba.__defaults__
# change default value of validate_features to False
XGBClassifier.predict_proba.__defaults__ = (None, False)

# plot
plot_partial_dependence(estimator=clf, X=X_train, features=[0, 1], feature_names=X_train.columns.tolist())
plt.show()

# reset default keyword argument values to original
XGBClassifier.predict_proba.__defaults = tmpdefaults