Custom loss function in H2O

Question

I am using H2O via R. I am trying to build random forest, XGBoost, GBM models to solve multiclass problem.

The model performance insights that H2O provides are great but as one of the success criterias I have my own custom function that scores the model accuracy when model is used to score a set of users say, validation set. Generally speaking, this function rewards the prediction and punishes the mis-prediction. Given that the target classes in this problem are ordinal categories, the punishment score depends on the number of places by which prediction was missed. The inverse of this custom function can also be thought of as a loss function.

I am wondering if and how I can plug such custom loss function into H2O while training models.

Answer 1

While there currently isn't support for a custom loss function (though there is a jira ticket to add it in), there is some support for a custom evaluation function, you can find more details here: https://github.com/h2oai/h2o-3/blob/master/h2o-docs/src/dev/custom_functions.md.