How to write a custom evaluation metric in python for xgboost?

Question

I would like to add the kappa evaluation metric to use in xgboost in Python. I am having trouble understanding how to connect a Python function with xgboost.

According to the xgboost documentation, a "User can add multiple evaluation metrics, for python user, remember to pass the metrics in as list of parameters pairs instead of map, so that latter ‘eval_metric’ won’t override previous one"

This has been raised in xgboost's github page for R but not for Python.

For example if the kappa function is:

def kappa(preds, y):
    # perform kappa calculation
    return score

How do I go about implementing it with xgboost? Specifing 'kappa' as a string in the eval_metric parameter results in XGBoostError: unknown evaluation metric type: kappa.

Likewise specifying the kappa method object results in XGBoostError: unknown evaluation metric type: <function kappa at 0x7fbef4b03488>.

How can a custom evaluation metric be used in xgboost in python?

Answer 1

Change your method to:

def kappa(preds, y):
    # perform kappa calculation
    return 'kappa', score

And use it with feval argument:

bst = xgb.train(params, dtrain, num_rounds, watchlist, feval=kappa, maximize=False)

When writing custom evaluation metrics remember about setting maximize argument. Setting it to true means that the algorithm is getting better with bigger score of the evaluation metric.