Why does XGBoost not show me the correct best_iteration and ntree_limit?

Question

I'm creating a binary classification model using XGBoostClassifier but I'm having some problems getting the right value to best_iteration and ntree_limit.

The code below is my custom evaluation metric:

def xgb_f1(y, t):
    t = t.get_label()
    y_bin = [1. if y_cont > 0.5 else 0. for y_cont in y]
    return 'f1', f1_score(t, y_bin, average='macro')

This is how I create and fit the Classifier:

classifier = xgb.XGBClassifier(n_estimators=10000)
classifier.fit(X_train, y_train, 
               eval_metric=xgb_f1, 
               eval_set=[(X_test, y_test)], 
               verbose=True)

These are some results XGBoost shows me during fitting:

[1007]  validation_0-error:0.181395 validation_0-f1:0.731411
[1355]  validation_0-error:0.183721 validation_0-f1:0.735139
[1396]  validation_0-error:0.183721 validation_0-f1:0.736116
[1426]  validation_0-error:0.182558 validation_0-f1:0.737302
[3568]  validation_0-error:0.186047 validation_0-f1:0.737557
[3791]  validation_0-error:0.184884 validation_0-f1:0.7378
[9999]  validation_0-error:0.210465 validation_0-f1:0.708715

And as you can see the best iteration is the iteration number 3791 due to the highest f1-score, but when I call classifier.get_booster().best_iteration it shows that the iteration number 9999 (the last iteration) is the best, but is not. And when I call classifier.get_booster().best_ntree_limit it tells me the best limit is 10000, but I don't think so, because it gets me a lower f1-score than lower iterations.

Answer 1

I think you should make use of the early_stopping_rounds param. However, you'll still get the last iteration's model. Check out the xgboost docs for the xgboost.XGBRegressior.fit() method, under the early_stopping_rounds it says:

The method returns the model from the last iteration (not the best one).

The workaround would be to create a new classifier after the first round of training and set n_estimators so that it stops exactly where it did previously.

classifier = xgb.XGBClassifier(
    n_estimators=10000,
    early_stopping_rounds=50
    )
classifier.fit(X_train, y_train, 
               eval_metric=xgb_f1, 
               eval_set=[(X_test, y_test)], 
               verbose=True)
classifier_new = xgb.XGBClassifier(
    n_estimators=classifier.best_iteration
    )
classifier_new.fit(X_train, y_train,
                   eval_metric=xgb_f1, 
                   eval_set=[(X_test, y_test)], 
                   verbose=True)