Feature Importance for XGBoost in Sagemaker

Question

I have built an XGBoost model using Amazon Sagemaker, but I was unable to find anything which will help me interpret the model and validate if it has learned the right dependencies.

Generally, we can see Feature Importance for XGBoost by get_fscore() function in the python API (https://xgboost.readthedocs.io/en/latest/python/python_api.html) I see nothing of that sort in the sagemaker api(https://sagemaker.readthedocs.io/en/stable/estimators.html).

I know I can build my own model and then deploy that using sagemaker but I am curious if anyone has faced this problem and how they overcame it.

Thanks.

Answer 1

As of 2019-06-17, Sagemaker XGBoost model is stored on S3 in as archive named model.tar.gz. This archive consist of single pickled model file named xgboost-model.

To load the model directly from S3 without downloading, you can use the following code:

import s3fs
import pickle
import tarfile
import xgboost

model_path = 's3://<bucket>/<path_to_model_dir>/xgboost-2019-06-16-09-56-39-854/output/model.tar.gz'

fs = s3fs.S3FileSystem()

with fs.open(model_path, 'rb') as f:
    with tarfile.open(fileobj=f, mode='r') as tar_f:
        with tar_f.extractfile('xgboost-model') as extracted_f:
            xgbooster = pickle.load(extracted_f)

xgbooster.get_fscore()

Answer 2

SageMaker XGBoost currently does not provide interface to retrieve feature importance from the model. You can write some code to get the feature importance from the XGBoost model. You have to get the booster object artifacts from the model in S3 and then use the following snippet

import pickle as pkl
import xgboost
booster = pkl.load(open(model_file, 'rb'))
booster.get_score()
booster.get_fscore()

Refer XGBoost doc for methods to get feature importance from the Booster object such as get_score() or get_fscore().