subsample, colsample_bytree, colsample_bylevel in XGBClassifier() Python 3.x

Question

I've spent a good deal of time trying to find out what these "subsample", "colsample_by_tree", and "colsample_bylevel" actually did in XGBClassifier() but I can't exactly find out what they do. Can someone please explain briefly what it is they do?

Thanks!

Answer 1

The idea of "subsample", "colsample_by_tree", and "colsample_bylevel" comes from Random Forests. In it, you build an ensemble of many trees and then group them together when making a prediction.

The "random" part happens through random sampling of the training samples for each tree (bootstrapping), and building each tree (actually each tree's node) only considering a random subset of the attributes.

In other words, for each tree in a random forest you:

1. Select a random sample from the dataset to train this tree;
2. For each node of this tree, use a random subset of the features. This avoids overfitting and decorrelates the trees.

Similarly to random forests, XGB is an ensemble of weak models that when put together give robust and accurate results. The weak models can be decision trees, which can be randomized in the same way as random forests. In this case:

• "subsample" is the fraction of the training samples (randomly selected) that will be used to train each tree.
• "colsample_by_tree" is the fraction of features (randomly selected) that will be used to train each tree.
• "colsample_bylevel" is the fraction of features (randomly selected) that will be used in each node to train each tree.