Problem: My situation appears to be a memory leak when running gridsearchcv. This happens when I run with 1 or 32 concurrent workers (n_jobs=-1). Previously I have run this loads of times with no trouble on ubuntu 16.04, but recently upgraded to 18.04 and did a ram upgrade.
import os
import pickle
from xgboost import XGBClassifier
from sklearn.model_selection import GridSearchCV,StratifiedKFold,train_test_split
from sklearn.calibration import CalibratedClassifierCV
from sklearn.metrics import make_scorer,log_loss
from horsebet import performance
scorer = make_scorer(log_loss,greater_is_better=True)
kfold = StratifiedKFold(n_splits=3)
# import and split data
input_vectors = pickle.load(open(os.path.join('horsebet','data','x_normalized'),'rb'))
output_vector = pickle.load(open(os.path.join('horsebet','data','y'),'rb')).ravel()
x_train,x_test,y_train,y_test = train_test_split(input_vectors,output_vector,test_size=0.2)
# XGB
model = XGBClassifier()
param = {
'booster':['gbtree'],
'tree_method':['hist'],
'objective':['binary:logistic'],
'n_estimators':[100,500],
'min_child_weight': [.8,1],
'gamma': [1,3],
'subsample': [0.1,.4,1.0],
'colsample_bytree': [1.0],
'max_depth': [10,20],
}
jobs = 8
model = GridSearchCV(model,param_grid=param,cv=kfold,scoring=scorer,pre_dispatch=jobs*2,n_jobs=jobs,verbose=5).fit(x_train,y_train)
Returns: UserWarning: A worker stopped while some jobs were given to the executor. This can be caused by a too short worker timeout or by a memory leak. "timeout or by a memory leak.", UserWarning
OR
TerminatedWorkerError: A worker process managed by the executor was unexpectedly terminated. This could be caused by a segmentation fault while calling the function or by an excessive memory usage causing the Operating System to kill the worker. The exit codes of the workers are {SIGKILL(-9)}
According to the official scikit-learn library, the n_jobs parameter is described as follows: The number of parallel jobs to run for neighbors search. None means 1 unless in a joblib. parallel_backend context.
The cause of my issue was that i put n_jobs=-1 in gridsearchcv, when it should be placed in the classifier. This has solved the issue.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With