What is the difference between objective
and feval
in xgboost in R? I know this is something very fundamental but I am unable to exactly define them/ their purpose.
Also, what is a softmax objective, while doing multi class classification?
The XGBoost objective function used when predicting numerical values is the “reg:squarederror” loss function. “reg:squarederror”: Loss function for regression predictive modeling problems.
The eval_metric parameter determines the metrics that will be used to evaluate the model at each iteration, not to guide optimization. They are only reported and are not used to guide the CV optimization AFAIK.
Objective
Objective
in xgboost
is the function which the learning algorithm will try and optimize. By definition, it must be able to create 1st (gradient) and 2nd (hessian) derivatives w.r.t. the predictions at a given training round.
A custom Objective
function example:link
# user define objective function, given prediction, return gradient and second order gradient # this is loglikelihood loss logregobj <- function(preds, dtrain) { labels <- getinfo(dtrain, "label") preds <- 1/(1 + exp(-preds)) grad <- preds - labels hess <- preds * (1 - preds) return(list(grad = grad, hess = hess)) }
This is the critical function to training and no xgboost
model can be trained without defining one. Objective
functions are directly used in splitting at each node in each tree.
feval
feval
in xgboost
plays no role in directly optimizing or training your model. You don't even need one to train. It doesn't impact splitting. All it does is score your model AFTER it has trained. A look at a example of a custom feval
evalerror <- function(preds, dtrain) { labels <- getinfo(dtrain, "label") err <- as.numeric(sum(labels != (preds > 0)))/length(labels) return(list(metric = "error", value = err)) }
Notice, it just returns a name(metric) and a score(value). Typically the feval
and objective
could be the same, but maybe the scoring mechanism you want is a little different, or doesn't have derivatives. For example, people use the logloss objective
to train, but create an AUC feval
to evaluate the model.
Furthermore you can use the feval
to stop your model from training once it stops improving. And you can use multiple feval
functions to score your model in different ways and observe them all.
You do not need a feval
function to train a model. Only to evaluate it, and help it stop training early.
Summary:
Objective
is the main workhorse.
feval
is a helper to allow xgboost
to do some cool things.
softmax
is an objective
function that is commonly used in multi-class classification. It insures that all your predictions sum to one, and are scaled using the exponential function. softmax
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With